Topic modeling of prose fiction: Model assessment and interpretability (the case of Russian short stories of the 1900s–1930s)

The article presents two experiments investigating the interpretability of results obtained from automatic topic modeling of literary texts, addressing the broader question of the appropriateness of applying this method to fiction. The relevance of this research is grounded in the successful application of topic modeling to specialized texts, contrasted with the challenges posed by the metaphorical language and thematic complexity of literary works. The study aims to determine how and to what extent the topic distributions produced by the model (story-topic correlations) reflect the thematic aspects of short stories. The research material consisted of 3,000 short stories written by 927 Russian authors, including world-renowned figures such as Nobel laureates I.A. Bunin and M.A. Sholokhov, Russian “classics” like A.P. Chekhov and M. Gorky, as well as lesser-known and nearly forgotten writers. During the study, several samples were generated for each of the three chronological periods: (1) the beginning of the 20th century (1900-1913), (2) the era of wars and revolutions (l9l4-1922), and (3) the early Soviet period. Each period was represented by three samples consisting of 100, 500, and 1,000 short stories. For each sample, models were constructed using the LDA algorithm with various preprocessing options. The evaluation of topic interpretability was conducted in two stages. The first stage aimed to identify which preprocessing steps yielded the best interpretability of the resulting model. The model trained on the corpus without any POS filtering exhibited the highest interpretability, with 25% of the generated topics deemed interpretable by experts. In the second stage, only those 24 topics that were unanimously considered interpretable by all three experts were further analyzed. During the second experiment, two experts read all 127 texts from the resulting sample and evaluated each topic on a three-point scale: (1) fully corresponds, (2) partially corresponds, (3) does not correspond at all. The experiments revealed that the experts identified a good correspondence between the text and the automatically assigned topic in 52% of the short stories, partial correspondence in 24%, while the remaining 24% of the stories appeared unrelated to the assigned topic. Thus, 76% of all interpretable topics demonstrated a meaningful connection to the content of the stories, beyond being merely statistically significant word clusters within the texts. These results are quite promising and suggest that topic modeling can be effectively applied to fiction, allowing researchers to accurately identify typical themes within a collection of short stories without needing to read them all. However, achieving these results requires a careful preliminary selection of the topics generated by the model, ensuring their semantic coherence, as done in the first stage of the experiment. The authors express their gratitude to the members of the Scientific and Educational Group of Interdisciplinary Philological Research of the National Research University Higher School of Economics in St. Petersburg, especially to A.S. Karysheva, E.O. Kolpashchikova, A.Yu. Moskalenko, I.A. Delazari, I.S. Zavyalova, for their active participation and expert assessment at different stages of the research.

Keywords

Authors

References