Interpretation, Argument, Evaluation. A Workflow for Assessing LLM-Generated Interpretations of Poetry

1. Introduction

Despite the increasing diversification of literary studies in recent decades, the interpretation of literary texts remains one of its central practices. This is evident not only in the prominence of chapters on interpretation in key introductory works, but also in empirical studies highlighting the prevalence of interpretive articles in scholarly journals.¹ This focus on interpretation contrasts with current trends in computational literary studies, where machine learning methods such as large language models (LLMs) have been employed primarily for text analytic questions, which typically involve classification problems such as genre attribution or sentiment analysis.² Classification, the task of assigning previously defined categories to instances, can also be understood as a subform of description that is grounded in a theory and/or taxonomy of the relevant domain – for example, a theory of literary genres. Interpretation, by contrast, draws on specific theories of meaning – such as intentionalist or anti-intentionalist approaches – and, often in conjunction with text descriptions, attributes meanings to texts or text elements. An illustrative example may clarify this distinction: Identifying Hugo von Hofmannsthal’s poem Mein Garten as a sonnet – on the basis of its adherence to the characteristic features of this formally well defined genre – relies on a genre theory but does not presuppose a particular theory of meaning. By contrast, claiming that the poem explores the opposition between art and nature presupposes a theory of meaning that enables the attribution of such semantic properties to the poem.

In this work, we take initial steps towards exploring how interpretations of literary texts generated by LLMs can be evaluated. Here, we refer specifically to instruction fine-tuned LLMs that operate according to the ‘prompt-and-generate’ paradigm³, enabling them to generate coherent outputs in response to open-ended textual prompts.

Any attempt to explore the potential of LLMs for generating literary interpretations must contend with a foundational characteristic of literary studies: There are different conceptualizations of what it means to interpret a text. These differing conceptualizations are associated with distinct standards by which actual interpretations are assessed or evaluated. Such standards are often contested within literary theory and are frequently described as theory-dependent. Nevertheless, to evaluate generated interpretations of literary texts in a consistent and transparent manner, an explicitly formulated standard is needed – ideally one that is accepted independently of specific theoretical presuppositions. In other words, a well-defined set of evaluation metrics is necessary to enable the assessment of LLM-generated interpretations in the first place.

To explore such a set of metrics and address the issue of the theory-dependence of existing practices of interpretation and evaluation in literary studies, we propose a method of generating interpretations that reduces them to their argumentative core. By ‘argumentative core’, we refer to the fundamental argumentative structure that underpins a literary interpretation, independent of its stylistic or rhetorical presentation. While the modes of textualization in literary interpretations vary depending on the approach, it seems largely undisputed that they involve argumentation.

Building on this notion of an argumentative core, this paper seeks to identify an evaluative framework – drawn from theoretical debates – and to select and refine criteria derived from this framework to test whether they are suitable for assessing the argumentative core of LLM-generated interpretations. Our primary objective, therefore, lies in the selection and refinement of evaluation criteria and in demonstrating that they can be applied in an intersubjectively consistent manner. By contrast, the actual evaluation of LLM-generated interpretations falls outside the scope of this study. We contend that a meaningful evaluation becomes feasible only when such criteria are explicitly defined and their application ensures a high level of consistency among evaluators.

In addition, we would like to highlight several further limitations of this study in order to prevent potential misunderstandings. First, we do not address the question of how LLMs generate meaning – nor how this process differs from human meaning-making – and what implications this has for their alleged ‘understanding of language’.⁴ For heuristic purposes, we adopt what we refer to as a pretense stance: We treat LLM outputs as if they were produced by intentional agents, while fully acknowledging that these models do not possess genuine mental states. This interpretive strategy allows for a pragmatically useful engagement with LLM-generated texts, particularly in communicative and evaluative contexts. Conceptually, this stance is grounded in an externalist view of meaning, which holds that meaning does not arise from internal mental representations but from social, pragmatic, and interpretive practices. Within this framework, linguistic outputs are treated as meaningful insofar as they can be situated within communicative contexts and interpreted through interaction. This perspective is compatible with a range of externalist positions in current debates on LLMs and meaning, including accounts following Dennett’s intentional stance – which legitimizes mentalistic attributions based on their explanatory utility rather than ontological commitments – as well as accounts of derived intentionality such as those proposed in Borg (2025) or Koch (2025).⁵

A second limitation pertains to our theoretical orientation within literary studies. Just as there is no single literary theory, there is no monolithic discipline of literary studies, but rather a plurality of approaches. Our perspective is rooted in a specific tradition – namely, the German-language debates on literary theory informed by analytical philosophy: analytical literary theory.⁶ This approach is not characterized by adherence to a specific method but is instead defined by its commitment to scientific standards, conceptual clarity, precise question formulation, and rigorous argumentation (Köppe 2008).

In summary, this paper introduces a workflow for generating and evaluating literary interpretations using large language models. We begin by outlining the theoretical background, offering a brief overview of key debates on interpretation within literary theory. Next, we present different evaluation models, from which we select one – the framework by Strube (1992) – and justify this selection. We then address the question of how literary interpretations – specifically of poetry – can be generated by instruction-following LLMs in a way that aligns with Strube’s criteria. To achieve this, we adopt an approach based on the argumentative reconstruction of interpretations, ensuring that the generated texts can be systematically evaluated. Therefore, we operationalize Strube’s criteria in detail. Subsequently, we also describe the construction of reference/training data based on the argumentative reconstruction of existing poem interpretations. This is followed by an outline of the experimental setup, the presentation of results, and finally, the conclusion, which includes a discussion of limitations and suggestions for future research.

2. Interpretation in Literary Theory and CLS

Interpretation is a central concept in literary studies.⁷ The term is used to describe both the act of interpreting and the written results of this act, which can refer to either a single statement about a literary text or a complete essay dedicated to an exegesis. However, the meaning of the term remains a subject of ongoing debate.⁸ Bühler (1999), for example, describes 17 different uses of the word ‘to interpret’ with regard to the exegesis of texts in German. This ambiguity arises from the fact that the meaning of the term varies depending on its context of use and is influenced by related concepts – such as meaning, text, and work of art – as well as the theoretical frameworks in which these are embedded. Given this complexity, we refrain from proposing a single, fixed definition of interpretation. Instead, we adopt a scheme developed by Göran Hermerén, which provides a structured way to capture its diverse uses. Göran Hermerén describes ‘interpretation’ as the following relation between five variables: “X interprets Y as Z for U in order to V” (Herméren 1983, 142). This scheme makes it possible to differentiate between types of interpretation based on the definition of the variables: Depending on which object is interpreted in which way and with which purpose, a different type of interpretation results. According to Hermerén, the different types of interpretation correspond to different criteria to determine their ‘correctness’.⁹

However, the correctness or truth of interpretation statements is only one of several criteria that can be used to assess interpretations. Literary theory has worked out numerous such criteria, which to some extent were and are always determined by their theoretical and theoretical-historical standpoint. In 1992, Werner Strube proposed a set of criteria for the assessment of interpretations based on the language use and dominant interpretational practices in literary studies.¹⁰ Strube draws on the distinction between ‘Auslegung’ (exegesis) and ‘Deutung’ (interpretation) in German: He understands ‘Auslegung’ as the use of a specific scheme to interpret parts of a given text. ‘Interpretation’, on the other hand, refers to the combination of several such schemes into a final interpretation that refers to the entire text. Based on this distinction, Strube identifies four dimensions of the given practice of interpretation in literary studies and outlines relevant assessment criteria: 1) the way in which literary texts are described in literary studies, 2) the exegesis of a text, from which 3) the interpretation of a text differs, and 4) the mode of argumentation. For each dimension, he specifies conditions for their successful or unsuccessful realization. For the description in disciplinary terminology of literary studies, these are accuracy, relevance and appropriateness; for the exegesis, plausibility and historical coherence; for the interpretation, specificity, integrity and comprehensiveness; and for the argumentative structure, coherence, unforcedness and freedom from contradiction. It is controversial in literary theory whether such criteria can be independent of the guiding theory and the overarching interpretative goals.¹¹ However, it should be noted that these debates primarily concern the application of the criteria to a complete interpretation and the practices associated with it, rather than to argumentative cores. Underlying these debates is the fundamental question whether there are assessment criteria that are valid in general and, if so, what these criteria might be.

The determination of generally valid criteria for assessing interpretations is closely tied to the problem of evaluating interpretations, or their arbitrariness. Lutz Danneberg has reconstructed this problem in the form of the following argument (Danneberg 1992, 15):

• If there are no acceptable (or justified) criteria for evaluating interpretations with respect to their validity claims, then interpretations cannot be be assessed in terms of their validity claims.

• If interpretations cannot be assessed in terms of their validity claims, then they are considered to be of equal evaluative rank.

• If interpretations are considered to be of equal evaluative rank, then their choice is arbitrary.

The question of the evaluation of interpretation is accordingly one of the central questions of literary theory.

Given the complexity of these issues and the central role of interpretation in literary studies, it may come as no surprise that, to the best of our knowledge, no attempts have yet been made to develop (quantitative) evaluation metrics for interpretations (generated by LLMs). However, there are first attempts to explore the range of possible criteria for such evaluations and the benchmarking of LLMs’ text-interpretive abilities. One example is Jannidis et al. (2025): Their study investigates how well contemporary LLMs can “understand” poetry by probing nine core aspects of literary analysis – from meter and rhyme to figurative language and meaning – across increasing levels of interpretive complexity. The authors show that while LLMs perform well on semantic and interpretive tasks, they struggle with formally grounded operations such as scansion, phonetic pattern recognition, and culturally sensitive context integration. This study positions itself as an exploratory first engagement with the problem of interpretation, and thus as an attempt to delineate the literary- and communication-theoretical foundations for future benchmarking of LLM-generated interpretive statements. By contrast, in the present paper we develop and evaluate an approach that integrates the generation and the evaluation of interpretive statements so closely that a genuinely human – and ultimately quantifiable – assessment of them becomes already possible.

In addition to this, there are experiments from literature didactics that take a different approach by refraining from developing and explicating precisely defined evaluation metrics. For example, Susteck and Perder (2023) use four canonical German poems to investigate the extent to which ChatGPT 3.5 can cope with writing tasks in high school poetry analysis. The authors found that ChatGPT performs very convincingly, particularly in the generation of interpretation hypotheses that “link texts with stereotyped, but often appropriate interpretation patterns due to their comparatively high degree of vagueness”(Susteck and Perder 2023, 12). Susteck and Perder’s approach differs from the one presented by us in that 1) the evaluation of the generated texts is based on high school objectives – such as summary, classification within an epochal context, topic definition, and form analysis – rather than on explicitly operationalized evaluation criteria derived from a specific literary theory framework, 2) the OpenAI online chat interface is used and not the API, 3) no batch prompting is used, but a dialogical-chatbot-interaction, and, 4) interactive prompting was carried out.

3. Generating and Evaluating Interpretations

Our general workflow is depicted in Figure 1. Starting with a poem and existing interpretations, we first derive three argument reconstructions. These reconstructions are then used as examples in the prompt provided to the LLM, which also contains a new poem. The LLMs receive the reconstructed argumentations only in the form of individual statements, without any information about (a) which of Strube’s levels they correspond to or (b) which argumentative function they fulfill.¹² The model then generates output in the same style (i.e., as the argumentative core of an interpretation). The generated outputs are subsequently evaluated through manual inspection.

Figure 1: Schematic depiction of the workflow in this paper.

In the following, we will explain this workflow in detail, discuss the rationale behind our choices, examine how it aligns with a specific form of evaluation, and justify why this particular form was selected.

3.2 Generation of Interpretations Suited for Evaluation

Literary interpretations consist of distinct components such as thesis statements, textual evidence, analytical reasoning, and contextualization. Evaluating such interpretations holistically risks conflating these components, making it difficult to determine which aspects of the interpretation meet the required standards and which do not. For this reason, it is necessary to isolate and evaluate individual components of the generated text in relation to specific criteria. With simple prompt-based generation, it is neither clear (a) which components of the output can or should be evaluated with regard to which criteria or, if such criteria exist, how the generated text should be broken down into its components so that these criteria can be applied to the corresponding components, nor (b) which literary-theoretical assumptions the LLM realizes during generation. To address these challenges, we adopt a procedure that already suggests a certain output structure via the prompt: the generation of argumentative cores of interpretations. This approach simplifies the isolation and evaluation of the individual components. From the perspective of actual interpretative practice in literary studies, this method may seem unconventional, as such practices typically do not adhere to rigid organizational or structural schemes for the interpretations they produce. Nevertheless, we believe that this limitation is outweighed by the possibilities to guide the generation in such a way that the output is structured to align with the expected levels of output components, thereby facilitating a systematic evaluation according to the selected criteria from literary studies.

From a machine learning perspective, it makes sense to enrich the prompts with such reference data in order to achieve the above-mentioned goals. In this case, these data should consist of existing interpretations of literary texts. To assess their influence on the generation process and to evaluate texts generated with their help, it is useful to extract their central components in a structured form. To achieve this, we draw on a common practice in dealing with and analyzing scholarly texts: the reconstruction of their central arguments.¹⁵

Such reconstructions of arguments necessarily go beyond the literal wording of the texts examined. Descher and Petraschka (2018) identify the following dimensions of argument-reconstruction to which this applies in particular: reformulations, the clarification of text elements that require interpretation, the addition of argumentation steps, the sequence of arguments and the choice of argumentation scheme. Accordingly, reconstructions of arguments are themselves highly interpretative.

The guiding principle in our reconstructions is a specific version of the principle of charity: the aim of reconstructing the strongest possible argument from the texts. To achieve better alignment with Strube’s evaluation criteria, we base our reconstruction of the arguments on Strube’s distinction between the three levels of interpretation: description, exegesis, and interpretation. Therefore we reconstruct only the first three levels of the potentially multi-level argumentation (Figure 2), whose lowest level consists of arguments, whose premises consist of textual descriptions (= P₁ - P₄), whose conclusions (= C₁ - C₂) themselves figure as premises (= P₅ - P₆) of the central argument of the interpretations (= C₃).

Figure 2: Reconstruction of an argument.

Reconstructions of the central arguments of an interpretation are not uncontroversial in literary studies. They are considered reductionist, as they ignore numerous aspects that can also be significant for the persuasive power of interpretations, such as their rhetorical and stylistic design (Albrecht and Danneberg 2021; Descher and Petraschka 2018). Reconstructions that focus on the core structure and theses therefore differ in several respects from those that are more closely aligned with the subject-specific culture and practice of interpretation in literary studies, such as those recently developed for the interpretation of canonical narrative texts in the German-speaking world as part of the ArguLit project at the University of Göttingen (Winko et al. 2024): While the latter strives for a “‘dense description’ of the argumentative contexts as well as the characteristics of the interpretative texts”, taking into account “above all the diversity and linguistic complexity of the means of representation used” and accordingly reconstructs them on “a hermeneutic basis”(Winko et al. 2024, 43), the method used here is far more economical, focusing on the basic argumentative structure of the interpretative texts analyzed. Accordingly, we refer to the results of our argument reconstructions in the following as ‘argument-like structured interpretation’.

It is important to note that such reconstructions serve only as proxies for the actual textual practices of literary studies, as they lack the detail and comprehensiveness of reconstructions that account for all dimensions of literary arguments. Nevertheless, we employ them for several pragmatic purposes. First, they allow literary interpretations to be distilled to their argumentative core, presenting them in a format that is both generic and comparable. This reduction makes them significantly easier to evaluate and contrast than the often rhetorically and stylistically complex source texts. Second, these reconstructions can be flexibly expanded with additional layers of information, such as the types of arguments employed, key textual passages, or meta-information, including the interpretation’s objective or its underlying literary-theoretical framework. This makes them a modular and adaptable format for analyzing and comparing interpretive practices.

4. Reconstructing Core Arguments

The experiments presented in this paper utilize reference data for the generation and evaluation of interpretations of poems. In the given case, reference data are interpretations of literary texts that are representative (not in a statistical sense) of the interpretation practices within the discipline. An interpretation can be considered representative if a) it is used in teaching or b) it is frequently cited. The former applies to texts from the Reclam publishing house. The Reclam Verlag is a German publishing house renowned for its pocket-sized editions of classic literature and accompanying interpretations. It plays a significant role in German education by making essential literary works accessible and affordable for students and readers.

At the end of the 1990s, Reclam published collections of interpretative texts on works by 12 canonical German-language poets as a series of collected interpretations of poems entitled Gedichte und Interpretationen. From this series, we have selected three interpretations: Jochen Schmidt’s interpretation of Friedrich Hölderlin’s Hälfte des Lebens(Schmidt 1984), Hans-Georg Kemper’s interpretation of Georg Trakl’s Im Winter(Kemper 1999) and Marco Meli’s interpretation of Gottfried Benn’s Der Sänger(Meli 1997). The guiding selection criteria were that (a) the poems analyzed should be described as comprehensively as possible in the context of the interpretation and (b) these descriptions should serve as premises in the actual interpretation.

To strengthen the influence of existing interpretations on the generation process and to establish a framework for the systematic evaluation of the generated texts, we employ argumentative reconstructions of these interpretations (see subsection 3.2). In reconstructing the arguments, we follow the recommended procedure in Brun and Hirsch Hadorn (2021) and supplement it with insights from Winko et al. (2024): The reconstruction process begins with a close reading of the text to be interpreted, followed by the development of a structured overview of the interpretation texts. Based on this overview, we identify the central thesis along with its supporting premises, reformulate unclear, incomplete, or inconsistent statements, and add missing premises or conclusions where necessary. In doing so, we balance two opposing principles: On the one hand, we aim to stay as close as possible to the original formulations; on the other, we seek to strengthen the reconstructed arguments to ensure they provide sound and coherent reasoning. This tension is particularly relevant when adding missing rules of inference.

Such domain-specific rules of inference are according to Winko et al. (2024, 263) “assumptions of interpreters […] that underlie the plausibilization of their interpretative hypotheses, but usually represent the general framework assumptions or rules of the game for the plausibilization of interpretative hypotheses as implicit presuppositions that are potentially shared by many representatives of the subject”. As implicit presuppositions, these rules are typically not explicitly formulated in the interpretation texts themselves and must therefore be supplemented in the reconstruction process.²⁰

When adding inference rules, we proceeded as follows: After isolating the main thesis of an interpretation and identifying its supporting premises, we first examined which inference rule could be supplemented with minimal intervention if no explicit rule was provided. In this process, we considered not only deductive arguments but also inductive reasoning and inference to the best explanation. For instance, if an interpretation argues – based on close readings – that the two stanzas of a poem stand in a relationship of allegory and reflection, we would reconstruct the argument as one from circumstantial evidence rather than explicitly formulating a deductive inference rule. Only when no inductive reconstruction was possible based on the given premises and conclusion we introduced a conditional inference rule, adhering to the principles of argumentation reconstruction outlined in Brun and Hirsch Hadorn (2021).

In the following we will use Hans-Georg Kemper’s interpretation of Georg Trakl’s poem Im Winter as an example to illustrate the procedure of argumentative reconstruction of an interpretation of a poem. Kemper’s interpretation is divided into five parts, three of which focus on a particular dimension of the poem: After a brief introduction (Kemper 1999, 43), in which Kemper articulates his three central hypotheses, the first part (pp. 44-48) is devoted to the description and exegesis of Trakl’s expressionistic sequential style (Reihenstil). The second part (pp. 48-55) examines the sound-symbolic, motivic and structural repetitions of the poem, while the third part (pp. 55-58) explores the intra- and intertextual references to the rest of Trakl’s lyrical oeuvre. These dimensions are brought together concisely in an overall interpretation (pp. 58).

Kemper opens his article with the following hypotheses: Trakl’s Im Winter belongs (1) “to the early examples of the expressionistic sequential style” and breaks with the characteristics of the classical and romantic tradition of German poetry. It simultaneously realizes (2) “the poetic design of a referenceable winter image of high sensual plasticity”, which, however (3) “through its sensual charge and connotative approximation of the motifs” leads these motifs “to lose their everyday linguistic meaning and an autonomization of the poetic texture setting in”.

Each of the following three sections is dedicated to the development and support of one of these three hypotheses. The first part explores in detail the realization of the expressionistic sequential style in Trakl’s poem and examines its consequences in relation to classical-romantic German poetry. The second part compares the poem with Bruegel’s The Hunters in the Snow to show that Trakl’s poem, like Bruegel’s painting, is characterized by “haunting plasticity and suggestiveness”. According to Kemper, the sound symbolism – especially assonances and alliterations – as well as motivic and structural repetitions contribute to this effect. In the third part, Kemper shows that “the multiplicity of the image parts and the approximation of the motifs […] promoted by the form causes a tendency towards the autonomization of the vocabulary that runs counter to its referentiality”. This multiplicity is a result of the Trakl-specific connotations established throughout his lyrical oeuvre. In his conclusion, Kemper unites these three lines of argument to assert that Trakl’s poem combines the “destruction of traditional poetic meaning and the construction of an autonomous world of signs typical of Trakl” in such a way that their opposition is “‘suspended’ in sense of a refusal of meaning”.

If one attempts an argumentative reconstruction of Kemper’s interpretation, one is confronted with a complex argument. Kemper’s main thesis – and thus the conclusion of this argument – is the assertion that Trakl’s poem ultimately eludes a clear specification of meaning through the interplay of different principles of representation and form. The justification of this thesis can be reconstructed as a four-part argument (shown in Figure 3–Figure 6 in Appendix 1), with each part consisting of subarguments that have their own intermediate conclusions. The conclusions of the first three main arguments then form the premises of Kemper’s central argument. However, the reconstruction necessarily adopts substantive theoretical assumptions – concerning, for instance, the integrability of meaning levels and the criteria of definable meaning – that are not independently argued for in the original text but are nonetheless carried over into the reconstructed argument without being made explicit.

5. Experiments

1 You are a literary scholar.

2 Interpret the following poem in an argumentative form.

3 - - -

4 ### Orientation:

5 Below you will find an example that serves only as a structural template, not as a content template.

6 Use the argumentative structure as a guide, but develop new arguments that refer exclusively to the new poem.

7

8 ### Example (structure template only)

9 Title: {title}

10 Example: {poem}

11

12 Interpretation (example): {interpretation

13 - - -

14 ### New Poem

15 Title: {title_x}

16 Poem: {poem_x}

17 - - -

18 ### Interpretation:

6. Conclusions

In summary, the workflow developed here for evaluating generated argument-like interpretations of poems appears robust overall, yet its reliability varies across argumentative levels. While the overall moderate to high percent agreement, with notable variation across categories, indicates consistent annotation behavior, the divergent correlations between PA and Likert evaluations show that agreement and evaluative judgments do not align uniformly across descriptive, exegetical, interpretative, and inferential operations. Rather than indicating uniform effects, the correlations suggest category-specific relationships between inter-annotator agreement and perceived quality that admit multiple interpretations. Particularly noteworthy is the positive correlation between PA and Likert scores in the exegetical category, which cautiously suggests that higher annotator agreement may coincide with higher perceived quality of exegetical operations, without implying a strong or universal alignment between agreement and evaluation. In contrast, in the rules-of-inference category, the negative correlation in combination with comparatively high PA values can be read as tentative evidence that instances of agreement, where they occur, tend to coincide with lower Likert scores, potentially suggesting consensual identification of weak or unconvincing rules of inference. Conversely, the near-zero correlation in the interpretative category, alongside a moderate level of agreement, and the mixed correlations observed in the descriptive layer indicate areas where automated interpretations introduce forms of instability that human annotators detect to varying degrees. Together, these findings demonstrate both the sensitivity of the argument-like interpretive framework and the differentiated reliability of automated interpretive outputs depending on the type of argumentative operation involved.

We also see great potential for critical self-reflection of practices of literary studies in the fact that the evaluation of generated interpretation forces researchers to make his/her guiding background assumptions of these practices explicit. A central role in this would probably be played by the analysis of inference rules, as these are the manifestation of framework assumptions that secretly guide interpretation but are rarely made explicit. By explicitly formulating these inference rules in generating interpretations as presented here, much can be learned about the practices of the discipline by evaluating them, without having to carry out the laborious work of partially or fully reconstructing existing interpretations in advance.

7. Future Work

Future work can build upon this study on various levels: For instance, a detailed comparison of existing catalogs of evaluation criteria in terms of their consistency and operationalizability could contribute to refining the approach presented here. Additionally, the number of argumentative levels considered during reconstruction could be gradually expanded to examine the extent to which LLMs can adequately transfer the argumentative structure to new texts at increasing levels of complexity. Furthermore, efforts could be made to improve the skeletal reconstruction of arguments using advanced techniques such as prompt-tuning. By refining prompt design, it may be possible to generate more accurate and nuanced representations of complex literary interpretations. Additionally, larger test datasets are required to ensure the robustness and generalizability of the findings. These datasets should encompass a broader range of literary-historical epochs and interpretive frameworks, enabling a more comprehensive evaluation of the methodology across diverse contexts. Building on such datasets, future work should also aim to evaluate not only the LLM-generated interpretations, but also the reference interpretations themselves, using the same set of criteria. Comparing these evaluations may yield valuable insights into differences in interpretive practice and argumentative structure between human and machine-generated interpretations. Moreover, if a sufficient amount of manually annotated evaluation data becomes available, a classifier could be trained to automate the evaluation process, thereby enhancing scalability and consistency in future assessments. Finally, it is essential to explore alternative reconstruction approaches that might offer different or complementary perspectives on literary argumentation, contributing to a more versatile and multifaceted framework for the analysis of interpretive texts.

8. Limitations

This study is subject to several limitations. First, the chosen form of reconstruction does not align with common textual practices in literary studies or established literary conventions of representation. As a result, the approach may appear overly simplistic and neglect the specific context of interpretation, such as the intended audience, purpose, or situational relevance. Second, the validity of the study is limited by the fact that the reference data used for comparison was not itself evaluated, leaving potential biases or inaccuracies in the reference data unaddressed. Third, the reproducibility of results poses a challenge due to the non-deterministic nature of large language models and the use commercial models via API. Even with identical inputs and prompts, outputs may vary, making consistent replication difficult. Fourth, the experiment was conducted on a relatively small dataset of 18 examples, limiting the statistical robustness and generalizability of the findings. Fifth, the results cannot be generalized across all LLMs, as the analysis was restricted to a single model, which may not fully represent the capabilities or limitations of other models in the same category. Finally, the study evaluated reconstructed arguments and interpretations rather than directly assessing LLM-generated interpretations. This indirect approach might not capture the full potential or limitations of LLMs in generating literary analyses directly, leaving room for further exploration in future research.

9. Data Availability

Data and code can be found here: https://github.com/AxPic/poem-int-eval. They have been archived and are persistently available at: https://doi.org/10.5281/zenodo.18166524

10. Acknowledgements

We would like to thank the reviewers as well as Janina Jacke and the participants of the General Meeting of the DFG Priority Programme “Computational Literary Studies” in Würzburg for their constructive feedback on earlier versions of this contribution.

11. Author Contributions

Axel Pichler: Conceptualization, Data curation, Investigation, Methodology, Validation, Formal analysis, Writing – original draft, Writing – review & editing

Martin Endres: Validation

Nils Reiter: Formal analysis, Writing – review & editing, dealing with emojis in references

Appendix 1: Reconstruction of the Core Argumentation of Kemper

• Premise 1 (= P₁): The poem consists of three four-line stanzas with different individual images from the natural and human world.

• Premise 2 (= P₂): With the exception of three enjambments, the end of the sentence and the end of the verse coincide in the poem, which reinforces the pauses between the images.

• Premise 3 (= P₃): The three enjambments only connect main clauses and do not break up sentences.

• Premise 4 (= P₄): The simple, uniform sentence structure supports the clear separation of the images.

• Intermediate Conclusion 1 (= C₁): The poem realizes a new poetic image in each verse.

• Material rule of inference (= R₁): If images in a poem are arranged as a strict sequence of independent units without syntactic dependence, this constitutes a sequence of independent individual images, which is characteristic of the expressionist sequential style.

• Intermediate Conclusion 2 (= C₂): The poem realizes the expressionistic sequential style.

• Premise 5 (= P₅): The expressionist sequence style suspends the level of symbolic meaning.

• Conclusion (= Final C): The poem has no symbolic meaning.

Figure 3: Reconstruction of Argument 1.

• Premise 1 (= P₁): The first and second verses of the poem show a repetitive but varied optical sequence of movement from the ground to the sky and from the sky to the ground and back.

• Premise 2 (= P₂): The first verse of the poem introduces two antonymic assonance groups (‘a’ and ‘ei’), which lead to a blending of optical and haptic perceptions.

• Premise 3 (= P₃) The contrasting and at the same time analogous perceptual values thus created are continued and intensified denotatively and connotatively in the following verses of the poem.

• Intermediate Conclusion 1 (= C₁): The poem is characterized by sound-symbolic, motivic and structural repetitions and correspondences.

• Material rule of inference (= R₁): If a poem exhibits a high degree of sound-symbolic, motivic and structural repetitions and correspondences, then it potentially creates an impression of iconicity analogous to painting.

• Intermediate Conclusion 2 (= C₂): The poem creates an impression of iconicity analogous to painting.

• Material rule of inference (= R₂): If a text achieves a painting-like iconicity and its images can be related to real winter scenes, then it offers a poetic winter image of high sensual plasticity that can be referentialized.

• Premise 4: = P₄) The images in the poem can be related to real winter scenes.

• Conclusion (= Final C): The poem offers a poetic winter image of high sensual plasticity that can be referentialized.

Figure 4: Reconstruction of Argument 2.

• Premise 1 (= P₁): Trakl uses a limited vocabulary of images and motifs, which he combines and varies in different poems.

• Premise 2 (= P₂): The recurring use of certain images and motifs in different poems creates a Trakl-specific intertextuality.

• Premise 3 (= P₃): The poem contains images and motifs that also recur in other poems by Trakl.

• Intermediate Conclusion (= C₁):The poem participates in the Trakl-specific intertextuality.

• Premise 4 (= P₄): In the poem, numerous inter- and intratextual relations between the image parts and an approximation of the motifs are present.

• Material rule of inference (= R₁): If, in a poem, numerous inter- and intratextual relations between the image parts and an approximation of the motifs are present, this tends to lead to an autonomization of its vocabulary.

• Conclusion (= Final C): The poem has a tendency toward an autonomization of its vocabulary.

Figure 5: Reconstruction of Argument 3.

• Conclusion of argument 1 (= C₁): The poem has no symbolic meaning.

• Conclusion of argument 2 (= C₂): The poem offers a poetic winter image of high sensual plasticity that can be referentialized.

• Conclusion of argument 3 (= C₃): The poem has a tendency toward an autonomization of its vocabulary.

• Premise 4 (= P₄): These three aspects constitute different, non-integrable semantic levels of the poem.

• Intermediate Conclusion (= C₄): The poem offers different meanings on different representational levels.

• Material rule of inference (= R₁): If a poem offers different meanings on different representational levels, then it eludes a clear definition of its meaning.

• Conclusion: (= Final C): The poem Im Winter eludes a clear definition of its meaning.

Figure 6: Reconstruction of Argument 4.

Appendix 2: Generated Argument-Like Interpretation of Hofmannsthal’s Manche freilich …, prompted with Meli’s Reconstruction, and Gerstl’s balance–balance, also prompted with Meli’s Reconstruction

The following sections show two LLM-generated argument cores that have been manually evaluated in this paper. We show the interpretation core with the highest agreement and the highest average Likert scores (Hofmannsthal 3), as well as the interpretation core with a high agreement and correspondingly lower Likert scores (Gerstl 3).

1 Die ersten beiden Strophen des Gedichts »Manche freilich …« stellen zwei gegensätzliche soziale Existenzformen einander gegenüber.

2 Die eine Gruppe muss „drunten sterben” (V. 1) und liegt „mit schweren Gliedern / Bei den Wurzeln des verworrenen Lebens” (V. 5-6), während die andere „droben” (V. 3) wohnt und bei „Sibyllen” und „Königinnen” (V. 8) sitzt.

3 Die räumliche Opposition von „drunten” und „droben” korrespondiert mit der Gegenüberstellung von „schwer” und „leicht”.

4 Die Wiederholung des Wortes „manche” bzw. „andre” etabliert eine dualistische Weltordnung.

5 Diese dualistische Weltordnung wird in der dritten Strophe problematisiert.

6

7 Ein „Schatten” (V. 11) verbindet die getrennten Sphären miteinander, und „die leichten sind an die schweren / Wie an Luft und Erde gebunden” (V. 13-14).

8 Die Naturmetapher „Luft und Erde” suggeriert, dass diese Verbindung existenziell und unauflösbar ist.

9 In der vierten Strophe wechselt das Gedicht von der allgemeinen Beobachtung zur subjektiven Perspektive eines lyrischen Ich.

10 Das lyrische Ich kann „ganz vergessener Völker Müdigkeiten” nicht „abtun” (V. 15-16) und vermag das „stumme Niederfallen ferner Sterne” nicht von seiner „erschrockenen Seele” fernzuhalten (V. 17-18).

11 Das lyrische Ich artikuliert eine existenzielle Verbundenheit mit dem Leiden anderer und mit kosmischen Vorgängen.

12 Die Unmöglichkeit der Abgrenzung deutet auf eine ethische Verantwortung hin, die das privilegierte Individuum nicht ablegen kann.

13

14 Die Schlußstrophe universalisiert diese Erkenntnis durch die Feststellung, dass „viele Geschicke” nebeneinander existieren (V. 19).

15 Das „Dasein” verwebt alle Schicksale „durcheinander” (V. 20), wodurch individuelle Existenz als Teil eines überindividuellen Zusammenhangs erscheint.

16 Das lyrische Ich behauptet, sein „Teil” sei „mehr als dieses Lebens / Schlanke Flamme oder schmale Leier” (V. 21-22).

17 Die Metaphern „Flamme” und „Leier” verweisen auf ästhetische Autonomie und individuelles Leben.

18 Die Negation dieser Metaphern bedeutet eine Absage an die Vorstellung einer in sich geschlossenen ästhetischen oder individuellen Existenz.

19 Das Gedicht »Manche freilich …« verhandelt das Verhältnis von privilegierter und unterprivilegierter Existenz sowie die Verantwortung des Individuums gegenüber dem kollektiven Leiden.

20 Wenn ein Gedicht soziale Gegensätze thematisiert und zugleich die Rolle des Dichters reflektiert, kann es als sozialethischer und poetologischer Text gelesen werden.

21 Das Gedicht »Manche freilich …« ist ein sozialethischer und poetologischer Text.

22

23 Das

 lyrische Ich gehört offenbar zur privilegierten Gruppe der „Leichten”.

24 Das lyrische Ich kann sich jedoch nicht von der Verbindung zu den „Schweren” lösen.

25 Diese unlösbare Verbindung wird nicht als Begrenzung, sondern als Erweiterung der eigenen Existenz dargestellt.

26

27 Das Gedicht präsentiert eine Poetik der Empathie und der kollektiven Verantwortung, die über ästhetischen Autonomieanspruch hinausgeht.

28 Das Gedicht »Manche freilich …« steht damit in der Tradition einer ethisch engagierten Dichtung, die soziale Gegensätze nicht ausblendet, sondern in die poetische Selbstreflexion integriert.

1 Die erste Strophe des Gedichts »balance - balance« verneint explizit, dass räumliche Distanz eine »lösung« für unbenannte Probleme darstellt (V. 1-4).

2 Die zweite Strophe wiederholt und erweitert das Motiv der räumlichen Entfernung, indem sie das »sitzen auf stufen weit weg« als Zustand der Schwebe charakterisiert (V. 5-9).

3

4 Das Gedicht »balance - balance« verhandelt das Verhältnis von physischer Ortsverlagerung und innerer Problembewältigung.

5 Die physische Distanz (»rom« vs. »wien«) wird als unwirksame Fluchtbewegung entlarvt, da die Probleme »in meinem kopf« verbleiben.

6 Das zentrale Motiv der »schwebe« (V. 8) korrespondiert mit dem Titel »balance - balance« und bezeichnet einen prekären Gleichgewichtszustand.

7 Dieser Schwebezustand wird durch zwei gegensätzliche Bewegungen konstituiert: die triviale Ablenkung durch »lästige papier-lüsterne ameisen« (V. 7) und die bedrohlichen »abstürze« (V. 9).

8

9 Das Gedicht arbeitet mit einer Gegenüberstellung von scheinbaren und tatsächlichen Abgründen: Die Flugzeuge schweben »über den so genannten wirklichen abgründen« (V. 11), während das lyrische Ich über »gar nicht unwahrscheinlichen abstürzen« schwebt (V. 9).

10 Die Qualifizierung der Abgründe als »so genannte wirkliche« (V. 11) stellt die Hierarchie zwischen äußerer und innerer Bedrohung infrage.

11 Die Flucht erfolgt zweifach: räumlich von Wien nach Rom und mental durch »wegschauen« (V. 12) auf »kleine käfrige buchstaben« (V. 13).

12 Das Schreiben selbst wird als Ablenkungsstrategie kenntlich gemacht, die Teil des prekären Gleichgewichts ist.

13 Das Gedicht »balance - balance« präsentiert einen Zustand permanenter Instabilität, in dem weder räumliche Flucht noch künstlerische Ablenkung eine genuine Lösung bieten.

14 Die konsequente Kleinschreibung und die parataktische Syntax verstärken formal den Eindruck der Desorientierung und fehlenden Hierarchie.

15 Die Wiederholung des Titelworts »balance« deutet auf die Notwendigkeit ständiger Ausbalancierung hin, nicht auf einen erreichten Gleichgewichtszustand.

16 Das Gedicht »balance - balance« verweigert jede Form von Transzendenz oder Auflösung und beharrt stattdessen auf der Unausweichlichkeit des prekären Schwebezustands als existenzielle Grundsituation der Moderne.