Embracing Complexity in Academic Performance Appraisal | Problems of Governance. 2014. № Том 7. Выпуск 10.

Embracing Complexity in Academic Performance Appraisal

The field of bibliometrics gained momentum after the proposal of the h-index in 2005. Managerial decisions in academia increasingly rely on such metrics to facilitate performance appraisal. An analysis of the dynamics in the research process yields incentives for misguided practices and fraud. The complexity of the publication system renders bibliometrics as a leverage point through the indirect interdependencies with academics' motives and the product of research outputs. The prevailing paradigm reinforces output maximisation rather than exploration and knowledge dissemination. Case studies of Diederik Stapel and Don Poldermans illustrate the potential dangers of current practice when balancing elements fail. Those can be strengthened by a structural change in the measurement of performance. Broadening the range of input factors to include approximations of novelty holds the potential to achieve fundamental changes in the academic process.

Download file

Counter downloads: 187

Keywords

novelty, fraud, Don Poldermans, Diederik Stapel, knowledge dissemination, misguided practices, output maximisation, publication pressure, systems thinking, complexity theory, research, h-index, academic performance appraisal, новизна, bibliometrics, management, мошенничество, ошибочная практика управления, стимулирование публикуемости, системное мышление, теория сложности, индекс Хирша, оценка научной деятельности, менеджмент, библиометрия

Authors

Name	Organization	E-mail
Klement Lukas	INCEND Ltd. (Ireland)
Mosterd Max CHarly	INCEND Ltd. (Ireland)

Всего: 2

References

Vaidya J.S. (2005) 'V-index: A fairer index to quantify an individual 's research output capacity', British Medical Journal [online] http://www.bmj. com/rapid-response/2011/10/31/v-index-fairer-index-quantify-individual-s-research-output-capacity (Accessed 15 October 2012).

van Kolfschooten F. (2012) Ontspoorde wetenschap, Uitgeverij De Kring, Amsterdam.

Woit P. (2007) 'My H-index: 33' A Quantum Diaries Survivor [online] 11 April. http://dorigo.wordpress.com/2007/04/11/my-h-index-33/ (Accessed 27 July 2012). УДК 352(075)

Stapel D.A. (2012) Ontsporing, Uitgeverij Prometheus, Amsterdam.

Stephan P. (2012) How Economics Shapes Science, Harvard University Press, Cambridge.

TavareA. (2012) 'Where are we with transparency over performance of doctors and institutions?' British Medical Journal, Vol. 345, No. e4464.

Tijdink J., Vergouwen A.C.M., Smulders Y. 'Publication culture and burn out among Dutch medical professors', http://www.socialsciences.leiden. edu/cwts/news/cwts-seminar-20120127.html (Accessed 26 July 2012).

Stapel D.A. (2011) 'Diederik Stapel: 'De druk is mij te veel geworden' Volkskrant [online] 31 October. http://www.volkskrant.nl/vk/nl/3184/opinie/ article/detail/3004921/2011/10/31/Diederik-Stapel-De-druk-is-mij-te-veel-geworden.dhtml?utm_source=scherm1&utm_medium=button&utm_ campaign=Cookiecheck (Accessed 29 July 2012)

Soler J.M. (2007) 'A rational indicator of scientific creativity', Journal of Informetrics, Vol. 1, No. 2, pp.123-130.

Sidiropoulos A., Katsaros D., Manolopoulos Y. (2007) 'Generalized Hirsch h-index for disclosing latent facts in citation networks', Scientometrics, Vol. 72, No. 2, pp.253-280.

Rousseau R., Ye F.Y. (2008) 'A proposal for a dynamic h-type index', Journal of the American Society for Information Science and Technology, Vol. 59, No. 11, pp.1853-1855.

Schreiber M. (2008) 'To share the fame in a fair way, h(m) modifies h for multi-authored manuscripts', New Journal of Physics, Vol. 10, pp.1-9.

Rousseau R., Jin B. (2008) 'The Age-Dependent h-Type AR(2)-Index: Basic Properties and a Case Study', Journal of the American Society for Information Science and Technology, Vol. 59, No. 14, pp.2305-2311.

O'Riordan M. (2011) 'Erasmus MC fires Poldermans; ESC reviews his work' theheart.org [online] 23 November. http://www.theheart.org/ article/1315171.do (26 July 2012).

Kerr S., Tolliver J., Petree D. (1977) 'Manuscript characteristics which influence acceptance for management and social science journals', Academy of Management Journal, Vol. 20, No. 1, pp.132-41.

McKitrick R. (2011) 'Bias in the Peer Review Process: A Cautionary and Personal Account', in Michaels, P.J. (Ed.), Climate Coup, Cato Institute, Washington DC, pp.71-98.

Kelly C.D., Jennions M.D. (2007) 'h-index: age and sex make it unreliable', Nature, Vol. 449, No. 7161, pp.403-403.

John L.K., Loewenstein G., Prelec D. (2012) 'Measuring the prevalence of questionable research practices with incentives for truth telling', Psychological Science, Vol. 23, No. 5, pp.524-532.

Kahneman D. (2012) 'A proposal to deal with questions about priming effects' [online] http://www.nature.com/polopoly_fs/7.6716.1349271308!/ suppinfoFile/Kahneman%20Letter.pdf (Accessed 3 February 2013)

Jin B.H., Liang L.M., Rousseau R., Egghe L. (2007) 'The R- and AR-indices: Complementing the h-index', Chinese Science Bulletin, Vol. 52, No. 6, pp.855-863.

Hirsch J.E. (2005) 'An index to quantify an individual's scientific research output', Proceedings of the National Academy of Sciences, Vol. 102, No. 2005, pp.16569-16572.

Helbing D., Balietti S. (2011) 'How to create an innovation accelerator', European Physical Journal-Special Topics, Vol. 195, No. 1, pp.101-136.

Frangopol P.T. (2005) 'The Hirsch index - a new scientometric indicator for the evaluation of the results of a scientific researcher', Revista de Chimie, Vol. 56, No. 12 pp.1279-1281.

Goodstein D. (2002) 'Scientific misconduct', Academe, Vol. 88, No. 1, pp.28-31.

Fowler J.H., AksnesD.W. (2007) 'Does self-citation pay?' Scientometrics, Vol. 72, No. 3, pp.427-437.

Fanelli D. (2011) 'Negative results are disappearing from most disciplines and countries', Scientometrics, Vol. 90, No. 3, pp.891-904.

Fanelli D. (2009) 'How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data T. Tregenza, ed. PLoS One, 4(5) [online] http://www.plosone.org/article/ info%3Adoi%2F10.1371%2FjournaLpone.0005738 (Accessed 9 August 2012).

Egghe L. (2008) 'Mathematical theory of the h- and g-index in case of fractional counting of authorship', Journal of the American Society for Information Science and Technology, Vol. 59, No. 10, pp.1608-1616.

Falagas M.E., Alexiou V.G. (2008) 'The top-ten in journal impact factor manipulation', Archivum Immunologiae et Therapiae Experimenta-lis, Vol. 56, No. 4, pp.223-226.

Epstein D. (2007) 'Impact factor manipulation', The Journal of the European Medical Writers Association, Vol. 16, No. 3, pp.133-134.

Egghe L. (2007) 'Dynamic h-index: The Hirsch index in function of time', Journal of the American Society for Information Science and Technology, Vol. 58, No. 3, pp.452-454.

Egghe L. (2006) 'Theory and practise of the g-index', Scientometrics, Vol. 69, No. 1, pp.131-152.

Dume B. (2005) 'How high is your h-index?', Physics World, Vol. 18, No. 9, pp.7.

Denning P.J. (1997) 'A new social contract for research', Communications of the ACM, Vol. 40, No. 2, pp.132-134.

De RondM. (2005) 'Publish or Perish: Bane or Boon of Academic Life?', Journal of Management Inquiry, Vol. 14, No. 4, pp.321-329.

Costas R., BordonsM. (2007) 'The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level', Journal of Informetrics, Vol. 1, No. 3, pp.193-203.

Cabrerizo F.J. (2012) 'h-index Bibliography' [online] http://sci2s.ugr. es/hindex/biblio.php (Accessed 19 October 2012).

Cabanac G. (2011) 'Accuracy of inter-researcher similarity measures based on topical social clues', Scientometrics, Vol. 87, No. 3, pp.597-620.

BuddJ.M., Coble Z.C., AndersonK.M. (2011) 'Retracted publications in biomedicine: cause for concern', pp.390-395.

Budd J.M., SievertM., Schultz T.M., Scoville C. (1999) 'Effects of article retraction on citation and practice in medicine', Bulletin of the Medical Library Association, Vol. 87, No. 4, p.437.

Bedeian A.G. (1989) 'Totems and taboos: Undercurrents in the management discipline', Academy of Management News, Vol. 19, No. 4, pp.1-6.

Bornmann L., Daniel H.D. (2005) 'Does the h-index for ranking of scientists really work?' Scientometrics, Vol. 65, No. 3, pp.391-392.

Ball P. (2005) 'Index aims for fair ranking of scientists', Nature, Vol. 436, No. 7053. Р. 900.

Batista P.D., CampiteliM.G., Kinouchi O., Martinez A.S. (2006) 'Is it possible to compare researchers with different scientific interests?', Scien-tometrics, Vol. 68, No. 1, pp.179-189.

Anderson, M.S. et al. (2007) 'The Perverse Effects of Competition on Scientists' Work and Relationships', Science and Engineering Ethics, Vol. 13 No. 4, pp.437-461.