Generation of educational texts in Russian as a foreign language using Russian chatbots: An attempt of automatized comparative evaluation

When didactic materials in foreign languages are created, text becomes a significant means of teaching. This leads to the need to select suitable texts, as well as subject them to qualitative and quantitative adaptation, i.e. transform them in order to make them suitable for solving educational tasks. Currently, methods and scenarios for the effective use of generative artificial intelligence are being actively sought, and the potential usefulness of chatbots based on it is being explored both for the methodology as a whole and for individual subject areas. Certain features of generative AI and chatbots based on large language models could mitigate a number of problems associated with the burden of selecting and adapting or independently creating educational texts in foreign languages. For example, chatbots based on generative AI are capable of generating texts based on sets of keywords, one of such possibly being a part of the lexical minimum in a foreign language. However, when a text is generated, arbitrary lexical and grammatical material can be added to the words of the lexical minimum, making it impossible to use it without adaptation at the target level of language proficiency. In order to establish the magnitude and significance of such deviations, an automated comparative study was carried out on a set of educational texts in Russian as a foreign language, generated through popular Russian public chatbots based on generative AI. For generation purposes, an elementary-level lexical minimum for Russian as a foreign language, divided into topical groups, is used. Based on these topical groups, small texts are created using GigaChat and Alisa (YaGPT) chatbots and then evaluated using a specialized automated analyzer called Tex-tometer. Results were obtained, according to which 50 texts of 25 topical groups have an average size of 1070.4 characters with spaces, an average level of "Early B1" (1st certification level of proficiency in RFL), an average coverage of the elementary-level lexical minimum equal to 71% and an average level of lexical diversity equal to 0.58 on a scale between 0 and 1. The Textometer's assessment, according to which the generated texts would be useful in studying 13 grammatical topics (9 of elementary level, 4 of 1st certification level of proficiency in RFL) was also taken into account. Notably, the average performance of the GigaChat chatbot is biased towards generating more complex texts, and that of the Alisa chatbot is inclined towards simpler texts. It is concluded that popular Russian chatbots intended for general use can be applied without further modifications for the purpose of generating educational texts on RFL of a certain level of complexity; this neutralizes or mitigates a number of problems related to selection and adaptation of educational texts, but at the same time does not solve these problems completely. The author declares no conflicts of interests.

Keywords

Authors

References