Operationalization and Interpretation Dependence in Computational Literary Studies

1. Introduction

The question of how computational literary studies relates (or should relate) to ‘traditional’ literary studies’ interests, theories and methods is a subject of constant debate (cf. e.g. Trilcke and Fischer 2016, 4–8 for an overview). This article is based on the assumptions that computational literary studies can and should in fact relate to core research interests of ‘traditional’ literary studies, and that it is worth going to some lengths to argue and demonstrate why and how this is possible. Since the interpretation of literary texts is one of the central concerns of literary studies and often poses particular challenges for computational approaches, this article discusses the relationship between the (computational) operationalization of literary studies concepts¹ for text analysis/classification and the property of interpretation dependence. Operationalization is defined here as the threefold task of developing an adequate intensional definition (i.e. a definition specifying the sense of the concept, usually by providing necessary and sufficient conditions, Cook 2009, 155), identifying the steps necessary to decide whether a concept applies to a text or text passage and carrying out this decision on a text or text corpus. Interpretation dependence is understood as a gradable property of statements about literary texts or of literary studies concepts. Its degree is defined by the extent to which the attribution of a concept to a text or a statement about a text depends (in terms of justification) on non-truth-preserving inference or (controversial) contextual assumptions. The main aim of the paper is to explain how (high-degree) interpretation dependence creates challenges for the operationalization of literary studies concepts and to suggest possible ways of dealing with these challenges.

The article is structured as follows: As the notions of operationalization and interpretation dependence are central to this article, I will briefly sketch the debates concerning these concepts in the fields of Digital Humanities/CLS resp. literary studies/analytical philosophy of literature and explain how my own definitions relate to previous definition attempts (subsection 2.1 and subsection 2.2). Based on these explanations, I will comment on the method I’m following in this paper, which combines theoretical (‘top-down’) considerations with pratice-based (‘bottom-up’) elements in a way that will need to be explained in more detail (subsection 2.3). The practice-based elements originate from the ongoing project to operationalize the narratological concept of unreliable narration (Jacke 2023a). Roughly speaking, this concept is meant to denote fictional narratives in which the narrator is not to be trusted, and it is often described as a highly interpretation-dependent concept (Kindt 2008).

The following sections are based on the different tasks that (according to my definition of “operationalization“) need to be carried out when operationalizing a concept: I will first discuss the task of developing an adequate definition (section 3), including an explanation of why this task might be perceived as being related to interpretation dependence, while I am arguing that it is, in fact, not. The next section will be devoted to the task of identifying the steps necessary to determine whether a concept applies to a literary text (section 4). In describing this task, I will also explain and illustrate the notion of interpretation dependence in more detail and show why and how a high degree of interpretation dependence significantly complicates this task. The next part of the paper focuses on the step of applying a concept to a text (corpus) – and sketches out ideas for dealing with potential problems (section 5). The paper concludes with a summary and an attempt to formulate general recommendations for operationalizing complex concepts coupled with a high degree of interpretation dependence (section 6).

2. The Central Concepts and Methods of this Paper

Before going through the different tasks that are necessary to operationalize a concept and analyzing how they are affected by the concept’s interpretation dependence, I would like to comment on and contextualize the concepts and methods that are central to my approach, starting with the notion of operationalization.

3. Finding a Definition

The first task of operationalizing a concept, i.e. finding an adequate intensional definition (usually by providing the necessary and sufficient features an object has to have to be an instantiation of the concept), is not always an easy one to begin with, as many concepts in literary studies lack explicit definitions. Typically, such concepts are only roughly described, with different features and variants of the concept being unsystematically mentioned and used in articles or books. It is often not made clear whether the characteristics mentioned are necessary or sufficient conditions⁴ of the concept, or whether they are, e.g., merely typical features. This problem is aggravated by the fact that concepts are often characterized merely by giving example cases, and it is not made explicit which features of the given case make it a part of the extension of the concept.⁵ One reason for this lack of explicit definitions in many literary studies publications may be that literary studies tends to approximate its meta-language to its object language (Fricke 1970), i.e. often attempts to make research publications aesthetically pleasing – and elements that are perceived as too formalistic (such as definitions) may undermine this aspiration.

Further confusion is caused by the fact that literary studies terminology is not necessarily used in a standardized way, meaning that the same term may have different definitions (and sometimes: different concepts, i.e. their extensions differ). This may be less of an issue with terminological neologisms (such as “homodiegetic”, Genette 2010, 158–159), but is very much an issue with literary studies terminology borrowed from natural/everyday language (such as “unreliable”, see below).

The task of developing an explicit definition (i.e. providing the defining features) for a concept is characterized by the challenge of balancing different criteria: Since we are usually dealing with pre-existing terms/concepts, we need to ensure that our definition is sufficiently similar to previous uses of the concept while at the same time avoiding some of the formal problems of previous definitions, such as lack of clarity. What we need are, basically, Carnapian explications (Carnap 1965, paragraph 2; Gerstorfer and Gius 2025) – but both the interpretation and weighting of his quality criteria (exactness, similarity, fruitfulness and simplicity) may be different (and especially challenging) for literary studies and other humanities disciplines (Jacke 2019, 290–298). In the context of the (computational) operationalization of a literary studies concept, it is particularly important to keep in mind that a definition should respect the relevant intentions and goals of literary studies in relation to the concept, and not maximize the chances of high inter-annotator agreement or automated recognition at their expense. Let me briefly elaborate on this assumption: While Carnapian explications usually generate concepts that deviate from previous uses to some degree, e.g. for the benefit of increased precision, neglecting the original literary studies intentions behind a concept will risk further estranging of traditional and computational literary studies. The key point here seems to be that, in the field of literary studies, the fruitfulness criterion is very heavily weighed: Literary studies concepts should help us to identify and talk about relevant, interesting (textual) phenomena, and those phenomena are often complex. To sacrifice fruitfulness for exactness thus goes against a crucial aspect of literary studies disciplinary self-conception. This is even more true if exactness comes in the form of maximized agreement, as textual ambiguity and a resulting pluralism of interpretation are also valued highly in literary studies. So if the maxim that literary studies intentions behind a concept should not be neglected in favor of maximized agreement is refuted, it should be for very good and well-argued reasons.

One way of dealing with this problem, i.e. of mediating between complexity/fruitfulness and exactness, is to use very inclusive definitions (i.e. definitions with a large extension) and/or disjunctive definitions (i.e. definitions that are organized as alternatives of subtypes if no common necessary and sufficient conditions for the general concept can be found).⁶ It is often advisable to select single types or sub-concepts of complex concepts for operationalization and make explicit how they relate to the complex concept as a whole or to other subtypes.

For the concept of unreliable narration, an inclusive and complex definition by weighing the quality criteria for explications in the context of literary studies has been proposed (Jacke 2019, 289–308). Since the resulting definition reveals the heterogeneity of the concept (we might actually be dealing with different concepts subsumed under one term after all), it has been decided to single out one subtype of unreliable narration for the attempt of the (computational) operationalization in the context of the CAUTION project (Jacke 2023a).⁷ This subtype can be called incorrect assertion, and it applies to a sentence of a fictional text if and only if in that sentence the narrator asserts a proposition about the fictive world of the text that is incorrect in that world.⁸ This excludes from the study, for example, cases in which narrators (merely) omit relevant elements of the fictive world, (merely) hold incorrect beliefs about the fictive world, as well as cases in which the narrators’ expressed (through words or actions) or internalized values are in conflict with the text’s message – all of which are often referred to as “unreliable narration”. Defined in this way, this (sub-)concept is suitable for use in the context of text analysis, as it can help to categorize sentences or text passages. To use it for text or narrator classification would require the definition of a grading system and/or threshold value to transform the analysis results into a synthesized label for a text or narrator.

It should be noted that the development of a definition might always be perceived as a reduction in that it does exclude some previous uses of the term in question (even in the case of the most comprehensive definitions that pay a lot of attention to previous uses and intentions). This careful reduction is a necessary step to ensure a fruitful use of terminology in academia. In the context of our project, we also have decided – among other reasons due to the heterogeneity of the concept of unreliable narration – to choose a sub-concept for operationalization. This may be a reduction on a practical level (as we are not trying to operationalize unreliable narration in its entirety), but not on a conceptual level (we are not suggesting that only false claims should count as unreliable narration.) Thus what can at best result from the project CAUTION can only be a partial operationalization of unreliable narration, though maybe a complete operationalization of the subconcept incorrect assertion.⁹

Now, the challenges outlined in this section might be seen as resulting from the interpretation dependence of the concept in question. For example, if two literary scholars disagree about whether a particular text passage is a case of unreliable narration, the reason for this may well be that they have different definitions of the concept in mind (Gius and Jacke 2017, 246–250) – yet such disagreement is often taken as an indicator or even proof of interpretation dependence. For the sake of clarity, however, I propose to distinguish between unclear concept definitions and interpretation dependence,¹⁰ the latter of which I will discuss in more detail in the following section.

4. Identifying the Steps – with a Special Focus on Interpretation Dependence

Having found or developed an appropriate definition and, if necessary, chosen from a variety of sub-concepts, the second task of operationalization will be to identify the various steps that are necessary to determine whether the concept applies. In the case of literary studies, the relevant questions are (1) what we need to know in order to decide whether a text or text passage exemplifies a concept and (2) how we can find out these things.

The first question can usually be answered if we have a good definition of the concept at hand. Take, for example, the definition of incorrect assertion: A sentence in a fictional text is a case of the unreliable narration subtype incorrect assertion if, and only if, in that sentence the narrator asserts a proposition about the fictive world of the text that is incorrect in that world. So to find out whether a sentence is a case of incorrect assertion, we need to know which proposition(s) about the fictive world the narrator asserts in it and what is true in the fictive world concerning the matter of those propositions.

The question of how we can find this out, however, seems to be a little more difficult. While part of the answer may be found in the definition, much of it depends on how exactly the phenomenon is implemented in the specific literary corpus, text or passage under study, or, more precisely, on the ambiguity or indeterminacy of the text with respect to the relevant questions. Thus, there may often be no general answer to this question – instead, the answer varies, for example, between genres or types of text, or even between individual texts and passages. So the best we can do may be to find parameterized answers that together cover most or many cases, which may lead to the identification of new, application-related subtypes of the phenomenon.¹¹

To illustrate this, let’s look at unreliable narration/incorrect assertion. The first question would be how to find out which proposition the narrator is asserting in a sentence. This may seem straightforward – but in a surprisingly large number of cases, phenomena such as incomplete sentences, rhetorical questions, syntactic or lexical ambiguity, as well as irony or figurative speech can make it difficult to identify which statement a narrator makes at a particular point in the text. To formulate rules for identifying propositions we would probably need to refer to, for example, Gricean conversations maxims (Grice 1975)¹² – but determining the appropriate level of detail when specifying rules for different subcases is a very challenging task.

The second question that needs to be answered in order to detect unreliable narration/incorrect assertion seems even more challenging: How do we find out what is true in the fictive world of a text? While there are theories and methods for answering this question, such as the reality principle or the principle of minimal departure (Lewis 1978; Ryan 1980), these theories do not seem to be of much help for many of the questions about fictive worlds that are typically relevant in the context of unreliable narration. For example, the reality principle/principle of minimal departure suggests that a fictive world resembles our world as much as possible, meaning that unless the text suggests otherwise, we should ‘import’ the knowledge we have about our world into the fictive world. However, in many cases of potentially unreliable narration, we have to choose between alternative possible events that could just as well happen in our world.¹³ On the other hand, if the choice is between a realistic and a supernatural possibility, the text often supports (roughly) equally well the different readings (an example would be Hoffmann’s Der Sandmann ([1816] 1994)); and if we are dealing with a world that contains supernatural elements to begin with, we cannot be sure of the extent to which the fictive world differs from ours. In general, finding out what is true in the fictive world of a text will require a complex consideration and selection of various textual features and extratextual assumptions and information, and writing this down as rules is far from trivial as we would again need parameterized or even individualized sets of rules, which seems hardly feasible.

This is a point where we need to be particularly careful when operationalizing complex concepts: Because it is often so difficult to pin down the steps that are necessary to decide whether a concept applies, there sometimes seems to occur a shift between steps 1 and 2 of operationalizing a concept – especially in the context of computational literary studies. Based on the assumption that some of the relevant features may be indeterminable or immeasurable, the ‘translation’ of a definition into a step-by-step measurement guide actually results in a new, pragmatic definition the extension of which is (probably) not identical to the extension of the original one and can be regarded as an approximation to the original concept. This practice is even more common when the addressee of this operationalization step is a computer rather than a human being (Pichler and Reiter 2021, 5), as the integration of non-truth-preserving inferences and extratextual assumptions (see below) in explainable computational models seems to pose a particular challenge. This kind of reductionism is problematic for reasons I have laid out in section 3 above: It increases the gap between traditional and computational literary studies.

In contrast to the issues associated with the definition of a concept (see section 3 above), the challenges that arise in trying to identify the steps necessary to determine whether a concept actually applies seem to be a consequence of the high degree of interpretation dependence of the concept and related statements about literary texts. To substantiate this claim – and to generalize from it – I will now talk about interpretation dependence in more detail, elaborating on some of the concept’s aspects that have been mentioned in subsection 2.2. As I have explicated there, interpretation dependence in the context of literary studies is understood here as a gradable feature of statements about literary texts (and of concepts used to make these statements).

Let us first consider statements about literary texts. A statement’s degree of interpretation dependence is determined by two factors (which may in turn be present in different degrees): its reliance on (controversial) extratextual assumptions and its reliance on non-truth-preserving inferences. To say that a statement relies on something presupposes the assumption that statements about literary texts are typically arguable, i.e. that we can provide arguments to support them (Descher and Petraschka 2019). An argument typically consists of premises and conclusions, and we use some kind of inference rule to get from the premises to the conclusion. Typically, there are a number of arguments that directly support a claim – and often it makes sense to also provide arguments that support individual premises, and so on. Thus the first factor determining the degree of interpretation dependence of a statement – its reliance on (controversial) extratextual assumptions – is a feature concerning the premises of the arguments supporting a statement. If it relies solely on textual features, this aspect factors into its degree of interpretation dependence with zero. If extratextual assumptions are required to justify the statement, the degree of this aspect of interpretation dependence is determined by the number of extratextual assumptions required, the number of controversial extratextual assumptions and their degree of controversy, as well as their relevance in the argumentation. The second factor that determines the degree of a statement’s interpretation dependence – its reliance on non-truth-preserving inference – is a feature that concerns not the premises but the inference rules/inferences used to get from the premises to the conclusion. If a statement can be deduced from the premises that support it, then this aspect factors into its interpretation dependence with zero.¹⁴ If non-truth-preserving inferences (such as induction or inference to the best explanation) are required, the degree of this aspect of interpretation dependence is determined by the number of such inferences, their inductive probability (lower probability means higher degree of interpretation dependence) and their relevance in the argumentation (cf. Figure 1).

Figure 1: Factors influencing a statement’s degree of interpretation dependence.

Now, none of the factors that make up the degree of interpretation dependence are usually easy to determine, and most of them probably can’t be determined exactly. In particular, aspects such as relevance or degree of controversy will often be a matter of individual judgment. It is also difficult to stipulate a way how the factors should be weighed against each other. If a statement about a literary text is based solely on textual features and deductive reasoning, it is not dependent on interpretation. However, depending on how many and to what extent the different aspects contribute more than zero to the interpretation dependence, the degree will vary between slightly above zero and an open maximum. But even if an exact calculation of the degree of interpretation dependence may not be possible (or not yet possible, see section 6 below), knowing the factors that determine it will help us to estimate its degree for certain statements and concepts and thus to understand operationalization problems better.

In addition to the degree of interpretation dependence, we can distinguish between different types of interpretation¹⁵ on which a statement about a literary text can be based. The first criterion constituting types of interpretation is the extent of the linguistic unit being interpreted – so basically the question is whether an interpretation applies only to a small part of a text (such as a sentence or even a word) or to the text as a whole. While this criterion, again, allows for a graded scale, we could call its poles ‘micro-’ and ‘macro-level interpretation’. Although in practice it can be difficult to determine the extent of the linguistic unit to which an interpretation applies, there seem to be relatively clear cases. To take up unreliability-related examples from above, it is a micro-level interpretation to decide whether a single statement by the narrator is meant ironically – even if some cotext is usually needed to make this decision –, but it is a macro-level interpretation to reconstruct the plot-building events in the fictive world the text is about. Because macro-level interpretations are usually far more complex than micro-level ones, relying on more assumptions and inferences, statements based on them tend to show a higher degree of interpretation dependence.

A second criterion that constitutes types of interpretation is the larger-scale interpretive endeavor (and the associated types of meaning) of which the statement about a text is a part. Interpretations of literary texts are very often either aimed at reconstructing aspects of the fictive world or something beyond it, e.g. instances of figurative meaning located not on the intrafictional communication axis (between narrator and their addressee) but on the extratextual axis of communication (between author and reader, cf. Schmid 2008, 48). These types of interpretation have previously been labeled ‘content-specifying’ and ‘content-transcending interpretation’ (Folde 2015). Both types can occur at both the micro and macro levels (cf. Figure 2). The previously given examples illustrate content-specifying interpretation on the micro and macro level respectively. An example of micro-level content-transcending interpretation would be the attribution of some kind of figurative meaning to a text/story element,¹⁶ while a macro-level interpretation of this type would involve deriving a generalized message from a text as a whole.¹⁷ Content-transcending interpretation tends to be more heavily interpretation-dependent than content-specifying interpretation, if only for the reason that the latter type usually relies heavily on text-based arguments (Tepe et al. 2009), while the former type relies on theories of interpretation and related contextual information/assumptions,¹⁸ which are also controversial in many cases.¹⁹

Figure 2: Types of interpretation.

Having discussed degree and type, the final aspect of interpretation dependence that I would like to comment on is the reason why a statement about a literary text can be interpretation-dependent. One possibility is that this reason is linked to a literary studies concept that is used to make a statement about a text. Variants of this reason are that the definition of the concept refers to elements that are necessarily related to interpretation dependence (e.g. when it refers to controversial contextual assumptions)²⁰ or that the definition refers to elements whose identification often or usually depends on non-truth-preserving inference or (controversial) textual assumptions. An example of the latter is the reference to the fictive world in the definition of unreliable narration/incorrect assertion: As a content-based (rather than form-based) concept, reconstructing the fictive world of a text often (but not necessarily) involves interpretation. Many concepts in literary studies have such a minimum degree of, or tendency towards, interpretation dependence. Narratological time categories, for example, usually require a reconstruction of the fictive events, their chronology and duration. Exceptions are some purely formal or stylistic concepts, some of which may even preclude interpretation dependence, e.g. if we analyze a text according to how colloquial its language is.

The second possibility is that the reason for interpretation dependence does not lie in the definition of the concept itself but in the makeup of the literary text, especially its potential ambiguity or openness (with regard to the phenomena in question). Some of the seemingly least interpretive concepts require interpretation when they are used to make statements about a text that is open or ambiguous in the relevant respects (cf. Figure 3).²¹ Again, this aspect may vary according to individual texts as well as individual text types. The example of unreliable narration/incorrect assertion is particularly interesting in this case: Typically, texts in which the narrator is potentially unreliable are constructed in such a way that makes statements about the fictive world (and thus statements about unreliable narration) highly dependent on interpretation. The particular type of text that falls into the category of literary fantasy is even defined by difficulties in identifying the relevant aspects of its world (oscillating between natural and supernatural readings, cf. Todorov 1972). On the other hand, however, in texts with a certain (application-related?) type of unreliable narration – namely resolved unreliability –, it seems not (or very little) interpretation-dependent to make the relevant diagnoses about the fictive world.²²

Figure 3: Reasons for interpretation dependence.

So, overall, unreliable narration/incorrect assertion seems to be a relatively high-degree interpretation-dependent concept (and statements about this type of unreliability of a narrator in a literary text are also relatively high-degree interpretation-dependent). This interpretation dependence seems to be the main reason why it is so difficult to identify the steps necessary to detect unreliability in a text. To formulate some general tendencies: Such problems seem to affect to a greater extent cases where there is a high degree of interpretation dependence, where the interpretation dependence is rooted in the definition of a concept and/or a (corpus of) highly ambiguous texts are analyzed, and where macro-level (especially content-transcending) interpretation is necessary.

How did we deal with the challenges posed by the relatively high degree of interpretation dependence of unreliable narration in CAUTION in the context of identifying the steps necessary to detect it? In general, when operationalizing a concept, the three different tasks (definition, identification of steps, decision making) are performed iteratively and inform each other (Gius and Jacke 2017 and Reiter 2020 for annotation-based operationalization). In CAUTION, this intermingling is particularly true for the second and the third tasks – and the idea of optimizing the step-by-step description of how to identify unreliable narration in a practice-based approach, i.e. by trying to make a (well-considered) decision about whether the concept applies, actually inspired a large part of the conceptual setup of the project, as I will explain in the following section. While our project to operationalize unreliable narration/incorrect assertion was not concluded yet at the time of writing this paper, I will give a few examples of how our attempt to decide whether the concept applies helped to refine the steps necessary for identifying it.

5. Deciding Whether a Concept Applies

CAUTION is designed as a three-tracked project, experimenting with different general approaches to operationalizing unreliable narration/incorrect assertion (or a reasonable approximation). In the following, I will introduce the core track, the deep track and the approximation track, explain how they interact and how they may help to gain further insights concerning the steps necessary for identifying our variant of unreliable narration.

The core track deals with unreliable narration as defined above in a ‘conventional’ annotation approach. While we do use annotation guidelines for this, the guidelines not yet attempt to detail the (parameterized) steps for identifying which proposition a narrator is asserting and how to reconstruct the fictive world. Instead, they mainly specify what an asserted proposition is (e.g. distinguishing it from evaluative utterances), which passages should generally be excluded from annotation (e.g. embedded speech), and how to select the annotation span (e.g. mainly subsentences with predicate, sometimes noun phrases).²³ Human annotators have been asked to annotate texts from a small corpus of German fictional narratives from the 19th century to the present. The corpus consists of four narratives that are usually considered unreliable (with different estimated degrees of ambiguity), four narratives that feature related phenomena (like improbable events or a highly personalized narrator) and one narrative without such features. The task is to annotate (sub)sentences in which the narrator makes an incorrect claim about the fictive world (“incorrect statement”) or might make such an incorrect claim (“undecided”). So the idea here is to operationalize unreliable narration as a category for text analysis. In addition to this, we experiment with using the concept to describe texts or narrators rather than individual (sub)sentences, as this seems to be the way concepts are often used in literary studies practice. Here, the annotators are asked to provide both a graded and an (almost) binary label for each text (in percentage and as “yes”/“no”/“undecided”-decision respectively). The annotation guidelines deliberately do not specify how such a label should be arrived at (apart from emphasizing that it should not be calculated simply on the basis of the number of annotations in the text). Also, the different types of annotations or categorizations are not the subject of discussion among annotators in this core track. The idea behind this is to collect some descriptive data on how such decisions turn out in practice – in the hope that data analysis will provide insight into possible regularities and relations. To increase the chances of success, we are also generating other types of data, as I will explain below.

In an attempt to optimize the annotation guidelines, i.e. the identification of steps for determining the target phenomenon as a part of operationalizing the concept for human addressees, we designed the deep track. It is aimed at analyzing the previous annotation decisions by, metaphorically speaking, taking a step back from the close reading and annotation of the texts and trying to develop and argue for a macro-level content-specifying interpretation for each text. In other words, the idea is to arrive at a coherent and well-founded reconstruction of the most relevant aspects of the fictive world for each text – which is one of two relevant steps for making the decision of whether unreliable narration in the form of an incorrect assertion occurs, as I have explained in section 4. In order to structure and systematize this complex task, and to make the results comparable, it is divided into the following subtasks: (1) identifying the relevant open questions about the fictive world, (2) identifying the possible competing answers to these questions, (3) selecting a preferred answer per question and (4) providing arguments for these decisions that are organized in the form of argument trees (Descher et al. 2023; Winko et al. 2024), for which visualization software such as MindMup²⁴ or Argdown²⁵ can be used. This task differs conceptually from annotation in that the data that is being generated is initially not directly linked to offsets of the text. Instead, what is being constructed is a proposition-based model of the text (or an important aspect of the text, namely the world it describes), which is systematically enriched with arguments that support the decisions leading to that model. Annotations (or offsets of the relevant text) may come back into play in the form of premises of arguments in the model, since (parts of) the text itself will be among the most important premises supporting content-specifying interpretation hypotheses of the text.

The connection between this experimental deep track and the overall goal of operationalizing unreliable narration is that it is intended to provide insight into the most obscure step in identifying unreliable narration – namely the reconstruction of the fictive world. The aim here is to get the former annotators to document and reflect on the decisions that influence their annotations, which they may have previously made mainly intuitively. In addition, an evaluative facet comes into play: Not only does the required way of documenting and providing arguments promote coherence and consistency more than annotation – the specific preparation of interpretation hypotheses and arguments also allows for a systematic and informed discussion among the former annotators, which may lead to a reconsideration of decisions in some cases. Ideally, this step will lead to both better informed annotation decisions and better step-by-step guidelines for identifying unreliable narration.

At the time of writing this paper, the former annotators have developed and discussed argument trees for two of the project’s corpus texts (Bendixen’s Meine falschen Eltern (cf. Bendixen 2015) and Schnitzler’s Andreas Thameyers letzter Brief, which were estimated to exhibit a low resp. medium degree of ambiguity with regard to the relevant questions about the fictive world) and revised their formerly intuitive annotation decisions concerning the occurrence of incorrect assertions on this well-reflected basis. The argument trees have not yet been systematically evaluated to extract more detailed instructions on how to reconstruct the relevant parts of a fictive world. However, here are some potentially interesing observations. First, for all trees (across texts and annotators), partial summaries or paraphrases of narrated content play important roles as premises,²⁶ and these summaries/paraphrases are in turn supported argumentatively by providing quotes from the texts.²⁷ The annotators tend to largely agree on which paraphrases and text passages are relevant in this context, especially on the most basic level(s) of the argumentation. Second, another common type of premise used to support arguments about the fictive world are psychological hypotheses. These are typically used to argue for hypotheses about characters’ motives for acting in a certain way. It is noticable here that annotators not only tend to cite different psychological literature – sometimes they also refer to different psychological mechanisms when explaining characters’ behavior.²⁸ Third, apart from psychological assumptions that serve as premises, annotators tend to base their arguments on different kinds of extratextual information,²⁹ the relevance of which is sometimes argued for by referring to theories of interpretation.³⁰ However, there were no cases in the argument trees in which theories of interpretation have in turn been supported argumentatively.

While a systematic evaluation of how exactly such results may help to refine the description of the steps necessary to identify unreliable narration in the form of incorrect assertion has to occur elsewhere, I would briefly like to mention two possibilities. First, if it turns out that one premise type occurs frequently and with high agreement between annotators, it could be advised to draw upon this type in the guidelines. Second, better knowledge about the possible contexts that are drawn upon when arguing for a reconstruction of a fictive world may help to stipulate specific contexts for a precise and narrower operationalization³¹ or to parameterize operationalizations.

The two tracks I presented so far are mainly aimed at operationalizing the concept of unreliable narration for human addressees. In contrast, the main aim of the approximation track is a computational operationalization of ‘something like’ unreliable narration – resulting in the automated detection of text features that are very often associated with unreliable narration. This track is based on the fact that literary studies research on unreliable narration often compiles lists of indicators that may hint that a text being unreliably narrated (Allrath 1998; Nünning 1999). These lists are usually very heterogeneous, ranging from linguistic text properties (e.g. exclamations) to narrator characteristics (e.g. emotional agitation). For our project, we have selected six potentially indicative narrator characteristics, some of which are assumed to correlate with specific linguistic text properties. Some of these indicators can already be detected relatively reliably with computational models (such as indicators of emotion (Klinger et al. 2020) – even though the underlying definition may differ in detail from the relevant unreliability indicators), others seem to be comparably easy to model, e.g. even in a rule-based approach (such as addressee orientation/awareness of a communicative situation). The idea is to apply such automated models to the corpus and to let human annotators detect them manually. This will not only allow a comparison between an automated and a manual detection of these indicators (as well as a possible refinement of the models based on the annotations) – it will also make it possible to analyze the actual relation of the indicators to unreliable narration/incorrect assertion in two ways (Jacke 2023b): by explaining whether and how the relevant indicators are logically related to unreliable narration, and by analyzing whether and to what extent the indicators actually co-occur with the detection of unreliability by the annotators. So far, we have concluded the manual annotation of the indicators and we have conducted first analyses of the relationship between manually annotated indicators and manually annotated cases of unreliable narration, which suggest that the selected linguistic indicators occur slightly more often in sentences that are categorized as incorrect assertion (Blessing et al. 2024a; Blessing et al. 2024b) – but further analyses are needed to substantiate and differentiate this claim. If successful, it may be possible to include references to certain indicators in the guidelines on how to determine the relevant aspects (uttered propositions and corresponding features of the fictive world).

The intended outcome of the approximation track is to develop a computational model that can automatically identify textual features that are likely to co-occur with unreliable narration. Another way of putting this would be to say that the approximation track is based on an alternative/pragmatic definition of a different concept of unreliability the extension of which is not identical to the concept of unreliable narration that is of direct interest to literary studies. The aim would be to use the data generated in the other tracks of CAUTION to analyze and explain as precisely as possible how the automated model relates to the original concept – this is a prerequisite for the model to be useful in literary studies contexts. In contrast to the other two tracks, the aim here is a (partial or approximate) operationalization not for humans but for computers, and it will be accompanied by instructions on how to work on with it in human operationalization, analysis and interpretation efforts.³²

In the following and final section of this paper, I will summarize the main points of the paper and attempt a generalization from the solutions adopted in CAUTION for operationalizing highly interpretation-dependent concepts and briefly sketch out ways of evaluating the different elements suggested in this paper.

6. Conclusion

This paper’s aims were to show how the operationalization of literary studies concepts may be complicated by a high interpretation dependence of these concepts and to suggest ways of addressing these problems to bridge the gap between traditional and computational literary studies. The theoretical ideas have been illustrated by drawing upon an example case: the ongoing attempt to operationalize the concept of unreliable narration. After having laid out the basics (definitions of “operationalization” and “interpretation dependence” as well as the interplay between theory and practice in connection with this paper’s aims), the paper has followed the steps of operationalizing literary studies concepts (i.e. defining the concept, identifying the steps, making the decision) with a focus on the implication of interpretation dependence in this process. The first step (that of developing a definition) is particularly difficult with literary studies concepts, as they typically aim to capture very complex and often heterogeneous phenomena and are neither introduced nor used in a standardized way. These problems are not directly related to interpretation (i.e. the act of making sense of a linguistic entity) but instead to conceptualizing. One suggestion for dealing with complex concepts is to select and weigh quality criteria for defining literary studies concepts and to make this reasoning explicit. In the case of very heterogeneous concepts, it is often useful to work with selected sub-concepts.

The second step of operationalizing literary studies concepts is to identify the steps necessary to determine whether the concept applies to a textual entity, which can be divided into two sub-steps: identifying what needs to be known in order to make this decision, and explaining how these things can be found out. While the first sub-step can often be carried out on the basis of the concept’s definition, the second is often very difficult and needs to be parameterized according to text types or even individual texts. Its difficulty seems to increase with the degree of interpretation dependence of the intended statements to be made about texts. The degree of interpretation dependence is determined by the extent to which a statement relies on (controversial) extratextual assumptions (taking into account the number and relevance of these assumptions plus the degree of controversy) or on non-truth-preserving inferences (taking into account the number, relevance and degree of inductive probability of these inferences). This degree increases when the concepts used to make a statement about a text are interpretation-dependent by definition, when the texts are ambiguous, when the statement concerns the text as a whole and aspects of meaning that go beyond a reconstruction of the fictive world.

While interesting ways of dealing with moderately interpretation-dependent concepts have been suggested (Pichler and Reiter 2022), a first recommendation of dealing with highly interpretation-dependent concepts (and their application to ambiguous texts) in the context of operationalization is to not try to strictly separate the second and third steps of operationalization (i.e. identifying the steps and performing the decision) but to use a collaborative and discursive setting of applying the concept/performing the decision as a means of gaining further insight into the often obscure steps. A second recommendation is not to attempt a full operationalization for computers but to go with a deliberate approximation/partial operationalization and to explain its relation to the original concept as precisely as possible.

Both of these recommendations are based on the acceptance that a full (and interpretable) computational operationalization of highly interpretation-dependent concepts is currently not possible. However, a cautious and modest approach to this issue from several sides, as suggested here, might make a full computational operationalization of highly interpretation-dependent concepts possible in the future: As many concepts – as well as the theories, methods and practices of interpretation – in literary studies are complex and opaque, most steps towards operationalizing highly interpretation-dependent concepts require more than just theoretical efforts. As suggested here, carefully designed practice-based collaborative studies, in which the assumptions and processes involved in applying a concept are thoroughly documented and deliberated upon, can pave the way to a better understanding of the relevant practices – which, in turn, is a prerequisite for the development of understandable computational models that serve the interests of literary studies.

To conclude this paper, I will briefly touch upon the question of how the most important theses and suggestions I developed in this paper can be evaluated. First, my suggested definitions of “operationalization” and “interpretation dependence” can be challenged by convincingly arguing that they don’t meet the quality criteria that I aimed to meet (i.e. sufficient degree of similarity with previous uses of the relevant terms, increased exactness, fruitfulness for traditional literary studies purposes), that the quality criteria should be weighed differently or that entirely different quality criteria should be aimed for. If my definitions are refuted, my argument is likely significantly weakened. Second, the usefulness of my analysis of the relation between interpretation dependence and operationalization problems can be questioned by showing that interpretation dependence can’t in fact explain the operationalization problems it claims to explain, or by showing that there are significant operationalization problems that can’t be explained via interpretation dependence.³³ Third, my suggestions for dealing with the operationalization problems can be questioned by arguing either that the suggestions are not feasible or that they don’t mitigate the problems they claim to mitigate. Forth and finally, how about the operationalization of unreliable narration I’m sketching out in this paper? As I spelled out in subsection 2.3, this specific operationalization attempt mainly serves to illustrate my theoretical thoughts but ultimately does not play a crucial part in justifying them. Also, the operationalization is difficult to assess because it is not finished yet. One thing I would like to point out, however, is that typical evaluation methods for operationalizations for humans or computers (like inter-annotator agreement resp. precision, recall, F1 score) in their current form have limited significance when it comes to highly interpretation-dependent concepts. Instead, a well-founded and discursively examined argumentation justifying annotation decisions may be a (complexity- and ambiguity-, hence literary studies-friendly, yet very time-consuming) way to go.³⁴ There is still some work left to do here.

7. Data Availability

As this is a theoretical contribution to the field of computational literary studies, this article does not rely on specific data. However, annotation data from the example project CAUTION used to illustrate some of its points is provided in connection with a previous project-related publication. The data can be found at: https://doi.org/10.5281/zenodo.10254506.

8. Acknowledgements

The CAUTION project is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number 449444411. Annotations have been carried out by Ann-Sophie Marie Ahnefeld, Johanna von der Fecht, Anina Karch, Marie Mader, Lena Schneider and Eva Schorn.

9. Author Contributions

Janina Jacke: Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing