Asymptotic properties and robustness of trimmed versions estimates of standard deviation and mean absolute deviations
In the work, various estimates of the scale parameter characterizing the “spread” of a random variable are studied. It is shown that traditionally used in practice estimates of the scale parameter, such as a sample estimate of the standard deviation, and an estimate of the average absolute deviations, have unlimited influence functions, and they are very sensitive to the presence of outliers in the sample. The paper proposes truncated versions of these estimates and, which are calculated not from the initial sample, but based on ordered statistics, from which the smallest and largest ordinal statistics are previously removed. These estimates are “protected” from the presence of outliers in the sample, they have limited influence functions and their characteristics depend significantly on the parameter a, which in practice leads to additional efforts to select this parameter. In the work, adaptive versions of these estimates are proposed, for which the parameter is determined based on the initial sample. The results of comparing estimates under the conditions of various observation models, in particular, under the conditions of the Gaussian model with large-scale contaminating, are presented. The results obtained lead to the following conclusion. In cases where there is no certainty that the initial distribution is Gaussian, or the sample may contain gross errors (outliers), it is more advisable to use the adaptive standard deviation or the estimate in the form of an estimate of the median of the absolute differences. These estimates have limited influence functions and, therefore, are “protected” from the presence of outliers in the sample, and they are preferable in terms of efficiency over the other estimates considered under the conditions of various observation models.
Keywords
scale parameter,
robust estimates,
outliers,
influence function,
adaptive estimatesAuthors
| Shulenin Valery P. | Tomsk State University | shulenin-vp@rambler.ru |
Всего: 1
References
Bickel P.J., Lehmann E.L. Measures of location and scale // Proc. Prague Symp. Asymptotic Statist. 1973. Prague Charles Univ. 1974. V. 1. P. 25-36.
Bickel P.J., Lehmann E.L. Descriptive Statistics for nonparametric models. I. Introduction. II. Location // Ann. Statist. 1975. V. 3. P. 1038-1044; 1045-1069.
Bickel P.J., Lehmann E.L. Descriptive statistics for nonparametric models. III. Dispersion // Ann. Statist. 1976. V. 4. № 6. P. 1139-1158.
Bickel P.J., Lehmann E.L. Descriptive statistics for nonparametric models. IV. Spread // Contributions to Statistics. Hajek Memo rial Volume / ed. by J. Jureckova. Prague : Academia, 1979. P. 33-40.
Шуленин В.П. Робастные альтернативы стандартному отклонению при обработке данных физических экспериментов // Известия вузов. Физика. 2016. Т. 59, № 6. С. 62-69.
Хампель Ф., Рончетти Э., Рауссей П., Штаэль В. Робастность в статистике. Подход на основе функций влияния. М. : Мир, 1989. 512 с.
Хьюбер П. Робастность в статистке. М. : Мир. 1984. 304 с.
Шуленин В.П. Робастные методы математической статистики. Томск : Изд-во НТЛ, 2016. 260 с.
Serfling R.J. Approximation Theorems of Mathematical Statistics. New York : Wiley, 1980. 371 p.
Serfling R.J. Generalized L-M-R-statistics // Ann. Statist. 1984. V. 12. P. 76-86.
Tarasenko F.P., Shulenin V.P. Connection of MD-estimates with classes of robust estimates of location parameter // 12th Prague Conf. Inf. Theory, Stat. Decision Functions, Random Processes. Prague,1994. P. 220-223.
Jureckova J. M-L-R-estimators // Handbook of Statistics / eds. P.R. Krishnaiah, P. Sen. Elsevier Science Publishers, 1984. V. 4. P. 463-485.
Hampel F.R. The influence curve and its role in robust estimation // J. Amer. Statist. Assoc. 1974. V. 69, № 346. P. 383-393.
Hampel F.R. Contribution to the theory of robust estimation : Ph. D. diss. Berkeley : Univ. California, 1968. 103 p.
Hampel F.R. A general qualitative definition of robustness // Ann. Math. Statist. 1971. V. 42. P. 1887-1896.
Janssen P., Serfling R., Veraverbeke M. Asymptotic normality for a general class of statistical functions and applications to measures of spread // Ann. Statist. 1984. V. 12, № 4. P. 1369-1379.
Janssen P., Serfling R., Veraverbeke M. Asymptotic normality of U-statistics based on trimmed samples // J. Statist. Planning and Inference. 1987. V. 16. P. 63-74.
Шуленин В.П. Асимптотические свойства GL и U- статистик // Вестник Томского государственного университета. 2004. Приложение № 9 (11). С. 184-190.
Shulenin V.P. Asymptotic Properties of the Trimmed GL-and U-Statistics // PRAGUE STOCHATICS’98 : Abstracts 6th Prague Symposium on Asymptotic Statistics, Prague, 1998, August 23-28. P. 84.
Hogg R.V. Adaptive robust procedures: a partial review and some suggestions for future applications an theory // J. Amer. Statist. Assoc. 1974. V. 69. P. 909-923.
Шуленин В.П. Свойства адаптивных оценок Ходжеса-Лемана в асимптотике и при конечных объемах выборки // Вестник Томского государственного университета. Управление, вычислительная техника и информатика. 2010. № 2 (11). С. 96-112.
Bickel, P.J. & Lehmann, E.L. (1974) Measures of location and scale. In: Proc. Prague Symp. Asymptotic Statist. 1973. Vol. 1. Prague: Charles Univ. pp. 25-36.
Bickel, P.J. & Lehmann, E.L. (1975) Descriptive Statistics for nonparametric models. I. Introduction. II. Location. Ann. Statist. 3, pp. 1038-1044; pp. 1045-1069.
Bickel, P.J. & Lehmann, E.L. (1976) Descriptive statistics for nonparametric models. III. Dispersion. Ann. Statist. 4(6). pp. 1139 1158.
Bickel, P.J. & Lehmann, E.L. (1979) Descriptive statistics for nonparametric models. IV. Spread. In: Jureckova, J. (ed.) Contribu tions to Statistics. Hajek Memorial Volume. Prague: Academia. pp. 33-40.
Shulenin, V.P. (2016) Robust alternatives to standard deviation in physical data processing experiments. Izvestiya vuzov. Fizika - Russian Physics Journal. 59(6). pp. 62-69.
Hampel, F., Ronchetti, E., Raussey, P. & Stael, V. (1989) Robastnost' v statistike. Podkhod na osnove funktsiy vliyaniya [Robust ness in statistics. Function-based approach influence]. Translated from English. Moscow: Mir.
Huber, P. (1984) Robastnost' v statistke [Robustness in Statistics]. Translated from English. Moscow: Mir.
Shulenin, V.P. (2016) Robastnye metody matematicheskoy statistiki [Robust Methods of Mathematical Statistics]. Tomsk: NTL.
Serfling, R.J. (1980) Approximation Theorems of Mathematical Statistics. New York: Wiley.
Serfling, R.J. (1984) Generalized L-M-R-statistics. Ann. Statist. 12. pp. 76-86.
Tarasenko, F.P. & Shulenin, V.P. (1994) Connection of MD-estimates with classes of robust estimates of location parameter. 12th Prague Conf. Inf. Theory, Stat. Decision Functions, Random Processes. Prague. pp. 220-223.
Jureckova, J. (1984) M-L-R-estimators. In: Krishnaiah, P.R. & Sen, P. (eds) Handbook of Statistics. Vol. 4. Elsevier Science Publishers. pp. 463-485.
Hampel, F.R. (1974) The influence curve and its role in robust estimation. Journal of American Statist. Association. 69(346). pp. 383-393.
Hampel, F.R. (1968) Contribution to the Theory of Robust Estimation. Ph. D. Diss. Berkeley, Univ. California.
Hampel, F.R. (1971) A general qualitative definition of robustness. Annals of Mathematical Statistics. 42. pp. 1887-1896. DOI: 10.1214/aoms/1177693054
Janssen, P., Serfling, R. & Veraverbeke, M. (1984) Asymptotic normality for a general class of statistical functions and applications to measures of spread. Annals of Statistics. 12(4). pp. 1369-1379. DOI: 10.1214/aos/1176346797
Janssen, P., Serfling, R. & Veraverbeke, M. (1987) Asymptotic normality of U-statistics based on trimmed samples. Journal of Statist. Planning and Inference. 16. pp. 63-74. DOI: 10.1016/0378-3758(87)90056-5
Shulenin, V.P. (2004) Asymptotic properties of GL and U-statistics. Vestnik Tomskogo gosudarstvennogo universiteta - Tomsk State University Journal. 9(11). Application. pp. 184-190.
Shulenin, V.P. (1998) Asymptotic Properties of the Trimmed GL-and U-Statistics. 6th Prague Symposium on Asymptotic Statistics. Prague. August 23-28. “Prague Stochatics'98”. Abstracts. p. 84.
Hogg, R.V. (1974) Adaptive robust procedures: A partial review and some suggestions for future applications an theory. Journal of American Statist. Association. 69. pp. 909-923.
Shulenin, V.P. (2010) Asymptotic and nonasymptotic properties of Hodges - Lehmann adaptive estimators. Vestnik Tomskogo gosudarstvennogo universiteta. Upravlenie, vychislitel'naya tekhnika i informatika - Tomsk State University Journal of Control and Computer Science. 2(11). pp. 96-112.
