Voluntary Internet-based testing as a source of valid group psychometric intelligence scores
Unproctored Internet-based testing (UIT) is becoming more popular in recruitment and employment settings due to its cost-effectiveness, and because it makes it possible to reach diverse and geographically distant populations. Growing body of evidences is accumulated about equivalence of UIT to traditional proctored paper-and-pencil testing as applied to surveys and personality inventories, but there is little information available about suitability of UIT for collecting ability-test data. We hypothesized that such data may be quite useful for estimation of psychometric intelligence of large groups differed on age, education level, regions of living and so on. To check this supposition we used large-scale data gathered via the site www.mil.ru from September 2012 to December 2017, which included 262652 protocols of carrying-out of cognitive screening test (CST). This UIT-data were compared to results of carrying-out this test by military conscripts sample (N = 3182) in controlled conditions in paper-and-pencil mode (CPPT-data). Tests of measurement equivalency included both traditional procedures (comparison of scores distributions, means, standard deviations and reliability indexes) and three more fine grained ones based on item-level responses (comparison of patterns of item's difficulties, item-scale correlations and item's loadings on g-factor). Although mean scores were slightly higher for integrated UIT-sample, this difference disappeared if education level of UIT- and CPPT -responders has been equated. Links of aggregated IQ-scores extracted from UIT-data with age and education had high level of similarity with analogous links calculated for sample of 27388 candidates equated for age and gender, which was tested by other cognitive test in controlled high-stake conditions. Indexes of congruency between different segments of UIT-data and between UIT-, CPPT- and computer-based data on scale- and item-level were compared with analogous indexes calculated for 15 control samples fulfilled five cognitive tests in controlled conditions in the same or different modes (total size more than 47000). Congruency between different segments of UIT-data were none the worse than between samples examined by the same tests in same-mode cases; congruency between UIT- and CPPT-data for the CST was not the worse than congruency between data obtained from samples examined by the same tests in cross-mode cases. Although percentage of repeated testing protocols in UIT-data was near 7.6%, retesting didn't bias the mean IQ-scores of educational groups (d<0.01). These finding suggests that UIT-data can be a source of valid information about group psychometric intelligence.
Keywords
неконтролируемое интернет-тестирование,
интеллект,
общая когнитивная способность,
психометрические характеристики,
измерительная эквивалентность,
факторный анализ,
Unproctored,
Web-based testing,
intelligence,
general mental ability,
psychometric properties,
factor analysisAuthors
| Ugonyaev Konstantin V. S | Institute of Psychology of Russian Academy of Sciences | skv-354@yandex.ru |
| Radchenko Yurij I. | Military Academy of General Staff | savaur@mail.ru |
| Sokolov Alexander A. | Military Academy of General Staff | as450@mail.ru |
Всего: 3
References
Шмелев А.Г. Практическая тестология. Тестирование в образовании, прикладной психологии и управлении персоналом. М. : Маска, 2013. 688 с.
Использование новейших информационных технологий в мероприятиях по профес сиональному психологическому отбору в Вооруженных Силах Российской Федерации : материалы научно-практической конференции (Москва, 2-3.10.2002 г.). М. : ГШ ВС РФ, 2003. 248 с.
Компьютерная психодиагностика в профессиональном психологическом отборе во енных специалистов РВСН : учеб.-метод. пособие / под ред. К.В. Сугоняева. М. : ЦИПК РВСН, 2005. 184 с.
Соколов А. А. Перспективы автоматизации профессионального психологического отбора в Вооруженных Силах Российской Федерации // 20 лет функционирования системы профессионального психологического отбора в Вооруженных Силах Российской Федерации, итоги и перспективы совершенствования : сб. мат. военно-науч. конф. (Москва, 22 апреля 2014 г.) / под общ. ред. С.В. Чваркова. М. : ВАГШ ВС РФ, 2014. С. 87-95.
Tippins N.T., Beaty J., Drasgow F., Gibson W.M., Pearlman K., Segall D.O., Shepherd W. Unproctored Internet testing in employment settings // Personnel Psychology. 2006. Vol. 59, № 1. P. 189-225. DOI: 10.1111/j.1744-6570.2006.00909.x.
Gosling S.D., Mason W. Internet research in psychology // Annual Review of Psychology. 2015. Vol. 66. P. 877-902. DOI: 10.1146/annurevpsych-010814-015321.
Soto C.J., John O.P., Gosling S.D., Potter J. Age differences in personality traits from 10 to 65: Big five domains and facets in a large cross-sectional sample // Journal of Personality and Social Psychology. 2011. Vol. 100, № 2. P. 330-348. DOI: 10.1037/a0021717.
Gebauer J.E., Sedikides C., Schonbrodt F.D., Bleidorn W., Rentfrow P.J., Potter J., Gos ling S.D. The religiosity as social value hypothesis: a multi-method replication and extension across 65 countries and three levels of spatial aggregation // Journal of Personality and Social Psychology. 2017. Vol. 113, № 3. P. e18-e39. DOI: 10.1037/pspp0000104.
Gosling S.D., Sandy C.J., John O.P., Potter J. Wired but not WEIRD: the promise of the Internet in reaching more diverse samples // Behavioral and Brain Sciences. 2010. Vol. 33, № 2-3. P. 94-95. DOI: 10.1017/S0140525X10000300.
Gleibs I.H. Are all 'research fields' equal? Rethinking practice for the use of data from crowdsourcing market places // Behavior Research Methods. 2017. Vol. 49, № 4. P. 1333-1342. DOI: 10.3758/s13428-016-0789-y.
Chuah S.C., Drasgow F., Roberts B.W. Personality assessment: Does the medium matter? No // Journal of Research in Personality. 2006. Vol. 40, № 4. P. 359-376. DOI: 10.1016/j.jrp.2005.01.006.
Vecchione M., Alessandri G., Barbaranelli C. Paper-and-pencil and web-based testing: The measurement invariance of the Big Five personality tests in applied settings // Assessment. 2012. Vol. 19, № 2. P. 243-246. DOI: 10.1177/1073191111419091.
Le Corff Y., Gingras V., Busque-Carrier M. Equivalence of unproctored internet testing and proctored paper-and-pencil testing of the Big Five // International Journal of Selection and Assessment. 2017. Vol. 25, № 2. P. 154-160. DOI: 10.1111/ijsa.12168.
International Testing Commission. International guidelines on computer-based and Internet-delivered testing: Version 2005 // International Journal of Testing. 2006. Vol. 6, № 2. P. 143-172. DOI:10.1207/s15327574ijt0602_4.
Crump M.J.C., McDonnell J.V., Gureckis T.M. Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research // PLoS ONE. 2013. Vol. 8, № 3. e57410. DOI: 10.1371/journal.pone.0057410.
Potosky D., Bobko P. Selection testing via the Internet: Practical considerations and exploratory empirical findings // Personnel Psychology. 2004. Vol. 57, № 4. P. 1003-1034. DOI: 10.1111/j.1744-6570.2004.00013.x.
Arthur W., Glaze R.M., Villado A.J., Taylor J.E. The magnitude and extent of cheating and response distortion effects on unproctored Internet-based tests of cognitive ability and personality // International Journal of Selection and Assessment. 2010. Vol. 18, № 1. P. 1-16. DOI: 10.1111/j.1468-2389.2010.00476.x.
Lievens F., Burke E. Dealing with the threats inherent in unproctored Internet testing of cognitive ability: Results from a large-scale operational test program // Journal of Occupational and Organizational Psychology. 2011. Vol. 84, № 4. P. 817-824. DOI: 10.1348/096317910X522672.
Радченко Ю.И., Сугоняев К.В. Разработка тестового комплекса для Интернет-самотестирования потенциальных кандидатов на военную службу по контракту // Актуальные проблемы психологического обеспечения практической деятельности силовых структур : сб. мат. 3-й Всерос. науч.-практ. конф. СПб. : С.-Петерб. им. В.Б. Бобкова филиал РТА, 2014. С. 317-324.
Grigoriev A., Ushakov D., Valueva E., Zirenko M., Lynn R. Differences in educational attainment, socio-economic variables and geographical location across 79 provinces of the Russian Federation // Intelligence. 2016. Vol. 58. P. 14-17. DOI: 10.1016/j.intell.2016.05.008.
Lynn R., Fuerst J., Kirkegaard E.O.W. Regional differences in intelligence in 22 countries and their economic, social and demographic correlates : a review // Intelligence. 2018. Vol. 69. P. 24-36. DOI: 10.1016/j.intell.2018.04.004.
Практикум по психодиагностике: конкретные психодиагностические методики. М. : МГУ, 1989. C. 112-126.
Hunt E. What makes nations intelligent? // Perspectives on Psychological Science. 2013. Vol. 7, № 3. P. 284-306. DOI: 10.1177/1745691612442905.
Российский стандарт тестирования персонала / Н.А. Батурин и др. // Организационная психология. 2015. Т. 5, № 2. С. 67-138.
Cohen J. A power primer // Psychological Bulletin. 1992. Vol. 112, № 1. P. 155-159. DOI: 10.1037/0033-2909.112.1.155.
McCrae R.R. A note on some measures of profile agreement // Journal of Personality Assessment. 2008. Vol. 90, № 2. P. 105-109. DOI: 10.1080/00223890701845104.
Furr R.M. A framework for profile similarity: Integrating similarity, normativeness, and distinctiveness // Journal of Personality. 2008. Vol. 76, № 5. P. 1267-1316. DOI: 10.1111/j.1467-6494.2008.00521.x.
Hausknecht J.P., Halpert J.A., Di Paolo N.T., Moriarty G.M.O. Retesting in selection: a meta-analysis of coaching and practice effects for tests of cognitive ability // Journal of Applied Psychology. 2007. Vol. 92, № 2. P. 373-385. DOI: 10.1037/0021-9010.92.2.373.
Kyllonen P.C., Zu J. Use of response time for measuring cognitive ability // Journal of Intelligence. 2016. Vol. 4, № 4. 14. DOI: 10.3390/jintelligence4040014.
Давыдов Д.Г., Чмыхова Е.В. Применение теста Стандартные прогрессивные матрицы Равена в режиме ограничения времени // Вопросы психологии. 2016. № 4. С. 129-139.
Регионы России. Социально-экономические показатели. 2017 : стат. сб. / Росстат. М., 2017. 1402 с.
