The (In-)Consistency of Literary Concepts. Operationalising, Annotating and Detecting Literary Comment

5.1 Feature Extraction

We preprocess texts with spaCy,¹¹ using its default tokeniser, part-of-speech (POS) tagger, lemmatiser and sentenciser for German and adding several custom preprocessing components:

• a dictionary-based normaliser that we trained on the German Text Archive¹² to account for spelling variants in older texts;

• the Universal Dependency parser, morphological analyser, clausiser and tense– mood–voice–modality tagger from Dönicke 2020;

• a direct speech tagger that recognises text between opening and closing quotation marks;

• a component that assigns Levin 1995’s categories to verbs and Hundsnurscher and Splett 1982’s categories to adjectives from GermaNet (cf. Hamp and Feldweg 1997);

• the sentiment tagger¹³ from Remus et al. 2010 as well as our own emotion tagger based on the NRC Word-Emotion Associated Lexicon¹⁴ (Mohammad and Turney 2010, 2013), which assign scores for positive/negative sentiment and Ekman 1992’s basic emotions, respectively, to each token.

Inspired by Dönicke 2021’s grammatical feature extraction for discourse segmentation, we extract features clause-wise from the clause, its noun phrases (NPs), the composite verb and free discourse elements (i.e. conjunctions, complementisers, sentential adverbs). Table 3 shows all features. Grammatical features have been found to work well for the identification of discourse segments – which are also a multi-clause-level phenomenon– in German (cf. Dönicke 2021) and might also include useful features for comment identification. For example, we expect verb categories such as tense or mood to be especially useful since a change in those often marks a narrative pause, as in (2). Here, the narrated time is interrupted, and the present tense in the first sentence changes to the past tense in the second one.

Punctuation is also integrated as feature. In (4), punctuation marks the direct speech, in which a comment is contained. We also integrate semantic categories for verbs and adjectives. Main verbs like lesen ‘read’ and verstehen ‘understand’ belong to Levin’s category of cognition, which we assume to be indicative for comments.

Since comment, especially attitude, can be expressed in an emotional manner, we include emotion and sentiment labels as features. (5) shows an excerpt for Einstellung that is highly expressive due to the usage of so-called ”thick concepts”, such as offen ‘expansive’ and wundersam ‘miraculous’, which ”combine evaluation and non-evaluative description” (Väyrynen 2021).

5.2 Classifiers

Since a comment passage spans an open number of clauses, we define a classification task at the clause level: When vectorising a text D = (c₁, …, c_n) with n clauses, we construct feature vectors ${\overrightarrow{c}}_1,\dots ,{\overrightarrow{c}}_n$ as described in subsection 5.1, which we then concatenate to context-sensitive vectors $X_D=({\overrightarrow{x}}_1,\dots ,{\overrightarrow{x}}_n)$ using a window of three clauses: ${\overrightarrow{x}}_i:={\overrightarrow{c}}_{i-1}\circ {\overrightarrow{c}}_i\circ {\overrightarrow{c}}_{i+1}$ . Given ${\overrightarrow{x}}_i$ , the classifier should predict all tags of passages that contain c_i. In a post-processing step, every maximal sequence of clauses with the same tag is combined into a passage, which is, however, not relevant for the evaluation, see subsection 5.3.

From our training set, we remove two texts as development set. To alleviate the impact of the split, we perform our experiments for three different splits as shown in Table 4. Since the median count of comment clauses per text is 205 (see Table 1), we take two texts with a total number of around 410 comment clauses in each split. Furthermore, split 1 uses two early texts, split 2 uses two late texts, and split 3 uses an earlier and a later text as development set.

In each split, we train (1) a multi-output classifier that consists of three independent binary classifiers (one for every subtag), and (2) a binary classifier that only distinguishes comment (any subtags) from non-comment (no subtag). As base classifier we use either (1) a decision tree or (2) a logistic regression, both with balanced class weights. Since the performance of a decision tree strongly depends on its maximum depth and minimum leaf size, we perform grid search on the development set to select the optimal values for these parameters (see Table 5), using the same values for all base classifiers. For the logistic regression, we optimise the solver and multi-class parameter, and the regularisation parameter C.¹⁶ During the grid search, we use (macro-averaged) F-score (cf. Sokolova and Lapalme 2009) as scoring function, which we also use for evaluation.

Additionally, we combine the classifiers from the three splits into one majority classifier. The majority classifier assigns those tags to a clause that are predicted by at least two of the incorporated classifiers.

5.3 Evaluation

Table 6 shows Precision, Recall and F-score for all settings. For the binary classifier, Precision measures how many of the clauses tagged as comment are also annotated as comment in the gold standard; Recall measures how many of the clauses annotated as comment in the gold standard are also tagged as comment. The F-score is the harmonic mean of Precision and Recall. For the multi-output classifier, Precision, Recall and F-score are calculated separately for each subtag first and then averaged.

Decision tree and logistic regression show similar results on both the development sets and the test set. The performance of both methods varies across splits, but the majority classifiers alleviate these discrepancies: In all but one setting, the majority classifiers achieve equal or better F-scores than the best of its incorporated classifiers.

Although the F-scores for decision tree and logistic regression are similar, Precision and Recall are not: The decision-tree classifier performs much better in terms of Recall at the cost of a lower Precision, whereas the difference between Precision and Recall is less extreme for the logistic-regression classifier.

Unsurprisingly, both methods achieve higher performance in the binary setting (54% and 59% for the majority classifiers) than in the multi-output setting (38% and 37% for the majority classifiers), where the classifiers have to distinguish subtags of comment.

5.4 Analysis

Somewhat surprisingly, every classifier performs better on the test set than on its development set. Part of an explanation might be that the test set includes more comment clauses than the development sets, see Table 7, and our classifiers are mainly driven by Recall. Table 7 also shows further differences between decision tree and logistic regression: With a logistic regression, the F-scores on the test set for each subtag are comparatively stable across training/development splits, whereas the decision tree’s F-scores show a greater variance. The majority classifiers achieve performance close to the best individual classifiers for each subtag, resulting in F-scores of 28% for Einstellung, 35%–36% for Interpretation and 48%–52% for Meta.

The comparatively high performance for Meta is outstanding, considering that Meta is the less frequent comment type in our data. In Table 8, we calculate for each subtag the average number of training clauses that contribute to one percentage point on the test set. We can see that the ratio is significantly lower for Meta (12) than for Einstellung (103), with Interpretation in between the two (60), something that illustrates that Meta is much easier to learn by our classifiers than the other comment types.

Our binary classifier is considerably better than the multi-output classifier. In general, it is not unusual that a classifier performs better for a binary tagset than a more differentiated one. Still, since we observed in the agreement that annotators tend to disagree between Einstellung and Interpretation while agreeing that a passage is one of both (see section 4), we trained an additional logistic-regression majority classifier that regards Einstellung and Interpretation as the same tag. (We left out all Meta passages for this.) This classifier achieves an F-score of 47% on the test set, which is 19% higher than that for Einstellung and 11% higher than that for Interpretation. Therefore, we assume that the difficulty of differentiating between Einstellung and Interpretation applies for both humans and machine-learning methods, whereas a joint category is easier to learn.

Table 9 exemplarily shows the most important features for one logistic-regression classi fier.¹⁷ Positive features are indicative for a subtag whereas negative features are indicative against a subtag.

Tense and mood/modality are learned to be relevant for all subtags. We have seen this in (3), where tense and mood shift from present indicative to past subjunctive to express a comment of type Einstellung. From the table, we can conclude that all three types often occur in past subjunctive, accompanied by different modal verbs (e.g. scheinen ‘seem’, lassen ‘let’, wollen ‘want’).

The comment types also differ in their presence within direct speech. While comments of type Einstellung frequently occur in direct speech, comments of type Meta rather occur outside direct speech. An explanation for this might be that utterances of characters in direct speech qualify for Einstellung, whereas Meta is mostly produced by the narrator. For Interpretation, the speech feature is not important. Instead, it is learned that comments of type Interpretation do rarely occur after quotation marks (» and «), which makes sense because they indicate a change of the speaker (from narrator to character or vice versa) and an interpretative comment typically follows a statement by the same speaker.

As anticipated in subsection 5.1, Example (5), a striking characteristic for Einstellung is the high importance of features related to emotion (Trust, Joy, and the more general feature Gefuehl ’emotion’). For Interpretation, we find that a subject in first person (I, we) or second person (you) is a negative indicator since only in third-person sentences something is told/interpreted about persons, incidents etc. For Meta comments, past tense is a negative feature. Instead, they often occur in grammatical future tense or with the modal verb wollen ’want’, which can also express semantic future. This is illustrated in (6), where we can also see the typical use of Kommunikation ‘communication’ verbs, such as erzählen ‘tell’.

In general, our classifiers tend to return many shorter comment passages, with interruptions between them, while we annotate longer passages in the gold standard. This is because we train the classifiers on the clause level, giving only three clauses as input, whereas human annotators can draw connections between clauses that are farther apart. We do not see this as a problem, as long as the relevant passages from a text are returned. (Example 7) compares the gold annotations (a) and the predictions by the logistic-regression majority classifier (b) for an excerpt from Wieland’s Geschichte des Agathon. For sake of illustration, Einstellung is boldfaced, Interpretation is italicised and Meta is underlined.

The excerpt is part of a long Meta passage in which the narrator reveals the conception of the main character Agathon and his confrontation with the sophist Hippias. The narrator outlines parts of the story, which can be seen as background knowledge that qualifies for an (overlapping) Interpretation passage. Both passages span several (≥ 8) sentences in the gold standard. The classifier detects shorter passages instead. It correctly recognises large parts of the excerpt as Meta. This is remarkable since the comprehension of the Meta passage is tied to its long context and our clause-based classifier is able to detect important parts of it. It also identifies large parts as Interpretation, but is missing the beginning and a short interruption. Lastly, it also identifies the Einstellung in the last part of the excerpt; as well as a short Einstellung passage which is not in the gold standard. The short passage is a good example for a false positive: It is probably labelled as Einstellung because it features the evaluative term bekannter maßen ’as is well known’, but it does not express attitude towards the diegesis and is therefore not an Einstellung comment in the gold standard.

Table 10 shows the most important features of the binary classifier. It mostly includes important features for Interpretation (see Table 9), which is the most frequent class in the training data. It also includes some important features for Einstellung, but important features for Meta are underrepresented, and there are even features with an opposite sign to those for Meta. This suggests that Meta passages are not individually learned by the binary classifier. This is not surprising when looking at Figure 1: Only 8% of all clauses in the training data are only annotated with Meta (other Meta clauses overlap with another comment type).

Figure 1

Overlap of comment clauses in the training data (left) and the test data (right).

6.1 General Considerations

As announced in the introduction, we do not consider the attempt to recognise literary comments as an end in itself. Rather, we want to use this example to illustrate that attempts to operationalise and recognise literary phenomena automatically can shedlight on the consistency of the concepts on which they are based.

When speaking of the ‘consistency of a concept’, we mean that (a) comparatively homogeneous phenomena fall under it and (b) that the concept is methodologically guiding in the sense that these phenomena are intersubjectively and automatically recognisable. Inconsistent concepts, on the other hand, describe comparatively heterogeneous phenomena, which cause hardly or not at all surmountable difficulties when trying to recognise them intersubjectively and/or automatically. Accordingly, ‘(in)consistency’ is a gradual concept: a concept can be more or less (in)consistent as the phenomena that fall under it have more or less relevant commonalities.

In the following, when we try to judge whether there is (in)consistency of a theoretical concept based on our observations on theory, operationalisation, annotation, and detection of it, we implicitly or explicitly carry out inferences to the best explanation. Generally, inferences to the best explanation have the following structure (see Lipton 2005; Bartelborth 2017, 200–291; here according to Descher and Petraschka 2019, 75):

• P₁: X is a fact that requires explanation.

• P₂: The hypothesis H₁ explains X.

• P₃: No competing hypotheses H₂, H₃, …, H_n explain X better than H₁.

• C: So H₁ is probably true.

Due to premise P₃, inferences to the best explanation are not truth-preserving, i.e., a true conclusion does not always follow from true premises. Even if one considers as many relevant alternative hypotheses as practically possible, one may simply miss a hypothesis that explains X better than H₁. Thus, claims about the consistency or inconsistency of certain concepts based on results of annotation and automation should be understood as hypotheses to be tested by further research.

In the case of comment, two main facts seem to need explaining:

• X₁: While two subtypes of comment (attitude comment and interpretative comment) can be annotated with little intersubjective agreement and detected with little success automatically, the opposite is true for meta comment.

• X₂: Automatic detection of comment by the binary tagger works well, although there is (to our knowledge) no robust overarching literary definition of comment and 2 of the 3 comment-subtypes are poorly recognised.

In the following, we first discuss the facts X₁, then X₂.

6.2 The (In-)Consistency of each Comment Type

One of our most intriguing findings is that annotation and automatic detection struggle with the same types of comment, although the annotation is based on a passage’s reading whereas the automatic detection is based on a passage’s form. Interpretative comment and attitude comment were annotated with only moderate and fair agreement and their detection also performed poorly with F-Scores below 40% and 30%. In contrast, meta comment achieves substantial agreement and can be detected well, with F-Scores close to 50%. Taking into account the relation between available training data for each comment-type and the performance of the multi-output-classifier, showed that meta comment is much easier to learn than the other comment-types (see Table 8; Meta performing better than Interpretation/Einstellung by a factor of 5 and 8.5). Do these surprising results suggest an inconsistency of the concepts ‘interpretive comment’ and ‘attitude comment’ as we operationalised them based on several theories of literary comment?

With respect to annotation, we suspect that mainly semantic properties (the occurrence of terms such as ‘narrative’, ‘truth’ or ‘invented’) of the meta comment passages are responsible for their good agreement. The vague terms ‘attitude’ and ‘interpretation’, on the other hand, made the development of precise annotation guidelines difficult. What was contentious in the discussion of concrete annotations was not only at what point something is an attitude or interpretation, but also where the difference between the two lies. For clarification, let us recall example (3) from Fontane’s Der Stechlin:

In this direct discourse passage the attitude of the main character Dubslav von Stechlin to a second marriage becomes clear. As an argument he uses the assertion that everyone more or less believes in the resurrection and the bad reputation of appearing with two wives. In the brackets between the direct speech, however, the narrator formulates a second attitude of Dubslav: he does not believe in the resurrection. Since this statement is not made by Dubslav but the narrator, we annotate it as Interpretation. Chatman attempts to distinguish between these two types on the basis of the judgment/evaluation-criterion. However, he leaves open at what point a statement is evaluative enough to be considered an evaluative ‘judgment’ rather than an interpretation. This turned out to be problematic. For example, is the use of a term like ‘eagerly’ sufficient to show that the speaker has a positive or negative attitude towards someone? We found that annotators, based on their reading impressions, answer such questions differently.

For automatic recognition, it is, among other things, the difference between interpretative comment and attitude comment that causes problems. If we train a classifier that treats Einstellung and Interpretation as one tag (binary classification without Meta), we obtain F-Scores that almost approximate the binary score (Einstellung +Interpretation +Meta). A problematic indicator of the attitude in both annotation and automatic recognition are ”thick concepts” such as eagerly or miraculous, which ”combine evaluation and non-evaluative description” (Väyrynen 2021).

If we exclude obvious alternative hypotheses such as unqualified annotators, inadequate machine-learning models, or errors in the statistical analysis of our classifiers,¹⁸ our findings suggest that (in contrast to meta comment) interpretive comment and attitude comment, as we have operationalised them, are not consistent concepts. The phenomena that fall under these two concepts are evidently too heterogeneous to be reliably recognised by humans and computers. Our findings on the automation side also suggest that there may be a consistent concept that encompasses all phenomena that fall under ‘attitude comment’ or ‘interpretive comment’. Defining this concept conclusively, without resorting (exclusively) to the vague terms ‘attitude’ and ‘interpretation’, would be a future task for Literary Studies.

6.3 The Inconsistency of the Generic Concept ‘Comment’

Although literary theory does not provide a consistent definition of the overarching concept of comment, our binary classifier (differentiating between comment and non-comment) achieves good results (F-Scores close to 60%). On the one hand, this is not very surprising because the binary classifier has (a) more training data per category than the multi-label classifier and (b) binary categorisation is less demanding. On the other hand, the classifier seems to accomplish the very thing that literary theory cannot provide (yet): a possibility to identify comment as a general phenomenon. What does this mean for a narratological concept of ‘comment’? We have already noted in section 3 that comment as a literary phenomenon is sometimes defined ex negativo. Therefore, many researchers refrain from defining comment and take an additive approach: Thus, ‘comment’ is understood as a set of related phenomena (phenomenon1 OR phenomenon2 OR …) whose commonalities are rarely discussed.

Our own approach takes a related route, by identifying three comment types that share narrative pause as common feature or prerequisite. Our proposal, having the following logical form: necessary feature AND (feature1 OR feature2 OR feature3), takes the form of what Fricke calls a “flexible definition” (Fricke 1981). However, we have seen that there is reason to believe that two of the criteria that our operationalisation of ‘comment’ uses (‘interpretative passage’, ‘attitude passage’ are themselves not consistent concepts (see section subsection 6.2). Thus, the question arises whether the generic concept ‘comment’ is a meaningful consistent literary category at all.

It is important to see that automatic detectability is no reliable indicator that there is an underlying consistent concept. Not everything computers can automatically recognise is based on a consistent concept. Suppose we define the concept ‘tapple’ as ‘being an apple or a table’. This would be a very inconsistent concept because the phenomena that fall under it have little in common except that they are material objects. Nevertheless, one could undoubtedly build a supervised model that recognises ‘tapples’; it would most probably use the features of apples on the one hand and the features of tables on the other. Please note, that this only prima facie contradicts what has been said on inconsistent concepts above. The difficulty with automatic recognition would be that the model would be highly susceptible to bias due to unbalanced training data: If the majority of the training instances are tables, apples will probably not be detected at all, because they share no relevant commonalities with tables.¹⁹

So how does our binary classifier work? Our comparison of the most prominent features between the binary classifier (comment vs. non-comment) and the multi-label classifier (Einstellung, Interpretation, Meta) yields an interesting result. 13 of the 20 most prominent features are features that also play a role for the recognition of the comment types (see Table 10). More importantly, only two of these 13 features are among the 20 most prominent features of all three comment types (subjunctive in the current or succeeding clause) and 9 features are indicative of one comment type only (according to the multi-label classifier). If the classifier had learned a general concept of comment, one would expect two kinds of features to dominate: features that are indicative of all three comment types and/or completely new features that played no role for the multi-label classifier. Therefore, our analysis suggests that the binary classifier, at least partly, uses feature combinations that are indicative of certain types of comment to recognise comment. The fact that 10 out of 20 most prominent features of the binary classifier are important features for interpretive comment (being the most common type in our training set, see Table 8), dovetails nicely with our expectation that a model that reflects a concept which is to a certain degree inconsistent is highly susceptible to bias due to unbalanced training data. Taken together, our results can be regarded as evidence for ‘comment’ being a rather inconsistent literary concept. The best explanation for the classifier not learning a general concept of comment is that the concept subsumes relatively heterogeneous phenomena that do not share enough relevant commonalities.

We have already underlined that our conclusions in the discussion section are ultimately hypotheses for which we have found some evidence, if we concede certain assumptions. There is one more background assumption that is relevant for our conclusion in this section of the discussion. Like many researchers in the Digital Humanities, we assume that literary phenomena manifest themselves at multiple levels (cf. Underwood 2019, 42), meaning that if there were a consistent narratological concept of ‘comment’, it would be reflected in linguistically available features. This assumption, rarely made explicit, may be more justified for an essentially textual phenomenon as comment than for phenomena that include relational properties. Let us suppose this background assumption is justified, so that our results show that ‘comment’ is a rather inconsistent concept. This would mean fundamentally re-examining the category of ‘comment’ and asking whether the important phenomena worthy of investigation that it describes cannot be grouped differently and/or partially subsumed under other concepts such as ‘authorial intrusion’ (Dawson 2016), ‘digression’ (Esselborn 1997–2003), ‘factual discourse’ / ‘serious speech acts in fictional works’ (Konrad 2017; Klauk 2015), ‘reflective passage’ (Gittel 2022), or Sentenz (‘aphorism’, Reuvekamp 1997–2003). At least this procedure seems appropriate to us, assuming that literary concepts should be also suitable for quantitative research nowadays.