We evaluate the following embeddings, which can be roughly categorized into more simple baselines on the one hand, and some more complex embeddings derived from sophisticated deep-learning language models on the other. The baseline embeddings are defined as follows:

The following deep-learning embeddings are derived from pre-trained static type-based embeddings:

Similarly, the following embeddings are derived from the output of BERT, which generates vectors for each token, but also takes into consideration the textual context of the entire input sequence.

In contrast, the following embeddings result from pre-trained language models following a Sentence-BERT-architecture, as proposed by Reimers and Gurevych 2019, 2020.

Evaluating the embeddings as described above requires us to formulate a task that probes each embedding space for its ability to represent certain dimensions of (dis–)similarity of poems via their distances in that particular embedding space, taking into consideration and comparing against the human annotations. As the embeddings define no particular distance function, we evaluate every embedding with each of the following three distance functions: Euclidean (L2), Manhattan (L1), and Cosine Distance.

We opted to replicate our prompts for the annotators by formulating a binary classification problem on a particular dimension of similarity, and checking if the model can replicate the majority vote. Note that these votes either take the value ‘annotated left’ or ‘annotated right’. Assume some embedding space and some distance function d fixed. For some annotated triple, let left, anchor and right denote the corresponding vectors in that embedding space. Now, we make the following prediction:

To compare the true majority annotations with the predicted ones, we use the balanced accuracy (arithmetic mean over the recall of both classes, cf. Grandini et al. 2020) as our metric. Note that the random ‘no skill’ classifier has a balanced accuracy score of 0.5.

We remark that variations on the above operationalization are possible as well, particularly if we do not omit cases where the majority of annotators chose ‘The middle text is equally (dis)similar to both the left and the right text’. However, while experimenting we observed that when including this third class ‘same’ in the operationalization, the balanced accuracy significantly drops. (We made the different balanced accuracies comparable by rescaling to the range 1/(1–#classes) to 1, so that performance at random scoring is always at 0.) We suspect that this difference in performance is caused by the complexity of the triples that were labeled with ‘annotated same’: Human annotators agree on the features which make them classify a text as ‘more similar to the focus text’. But the label ‘same’ is given when neither of both comparison texts shows obvious similarities to the focus text, but that does not imply that the comparison texts have any features in common; they can be different to the focus text in very diverse ways.

In particular, we experimented with the following two variations of the original operationalization:

As outlined above, variant (a) is solved with lower balanced accuracy than the original operationalization throughout all embeddings and variant (b) with even lower accuracy.

The results show for all dimensions except ‘form’ a clear increase with the complexity of text representation: Word Embeddings are better than sparse representations – with dynamic embeddings based on BERT showing a better performance than static embeddings – and sentence embeddings are better than word embeddings. The best sentence embedding is showing an acceptable performance, especially if the cosine is used. As almost all the strategies of text representation, which we applied here, have been developed with the main focus on the semantic aspect, it is not too surprising that the best model is the best in all dimensions. The one big exception is form. Using only a very small set of features is enough to match the annotations. Discussions with the annotators revealed that they usually based their decision on a very small set of observations.

The best model is paraphrase-mpnet. To evaluate all German sentence embedding models,11 which are available at this moment, we use the about 9,000 sentences of the Sick dataset (Marelli et al. 2014) which we had translated into German with DeepL. Our results show paraphrase-XLM-R (correlation with human annotations: 0.82) slightly ahead of paraphrase-mpnet (0.8165), which is why we include these two models and the best model based on static word embeddings (FastText-mean) in the next step.

To adapt the text representations to the specific textual dimensions (content, form, style, and emotion), we additionally apply similarity learning. The goal of this step is to learn a transformation of the vectors presented in the previous chapter that allows for better reproduction of the annotation. We use a siamese neural network (Bromley et al. 1993) for this purpose, which we modeled following the maaten network structure from (Szubert et al. 2019). The base model consists of three dense layers (500, 500 and 2,000 neurons) each followed by a normalization activation function (see Klambauer et al. 2017) and dropout. The input for the network consists of our annotated poem triples.

Regardless of the original size of its vector representation, each poem is transformed into a space with 128 dimensions. The loss, and hence the optimization objective of the network, is to maximize the difference between the focus text and the negative example while also minimizing the difference between the focus text and the positive example, i.e. the text which has been annotated as being more similar to the focus text. In short: in Euclidean distances (dist(anchor, negative) - dist(anchor, positive)).12 Learning rate decrease is bound to a reduce on plateau mechanism, which leads to strong performance gains compared to more common choices like constant or time-based decrease rates. The network’s performance is measured via the amount of correctly identified positive examples (accuracy, see Table 1).

In the following, we continue a discussion in German literary studies about the relationship of poems of realism to those of early modernism and the special position of naturalistic poetry in this development. We hope to contribute to this discussion by a mix of explorative methods and hypothesis testing. To enable the latter, we will condense positions in the debate into three hypotheses related to this transformation.

Hypothesis 1: The poetry of naturalism, as represented in the anthology Moderne Dichter-Charaktere, is predominantly traditional rather than modernist. The question of where exactly naturalism can be located between realism and modernism has been debated many times. In this context, the anthology Moderne Dichter-Charaktere, which is part of our corpus, is considered central to naturalist poetry. The anthology’s introductions emphatically assert the novelty and revolutionary character of the texts (especially Conradi 1885, pp. I–III). Research, on the other hand, is mainly of the opinion that these statements are exaggerated and that the poetry of the anthology is, on the whole, traditional (Vietta 1992, p. 194; Fähnders 1998, pp. 36–37; Sprengel 1998, p. 621; Austermühl 2000, pp. 350–51; Lamping 2000, pp. 145–146; Andreotti 2014, p. 17). However, some scholars, even if they consider the anthology as a whole to be traditional, argue that it was at least innovative in terms of content since new themes such as ‘big cities’ or ‘social issues’ were addressed (e.g. Fähnders 1998, pp. 36–37).

Hypothesis 2: Modernist poetry is heterogeneous, that is, more heterogeneous than realist poetry. While the poetry of realism, or at least the mass-produced poetry of this period, is considered by researchers to be relatively homogeneous (e.g Stockinger 2010, p. 88), modernist poetry is highly diverse, according to many scholars, given the simultaneity of a wide variety of literary movements (Anz 2007, pp. 330–331; Becker and Kiesel 2007, p. 30; Fähnders 1998, pp. IX, 4). But the hypothesis of modernist heterogeneity has its limitations. For example, some researchers support the view that modernism is homogeneous at least insofar as it responds to the same social-cultural problems (Vietta 1992, pp. 30–31:; Fähnders 1998, pp. 9–10; Becker and Kiesel 2007, p. 30; for further statements on the homogeneity of modernist poetry see H. Friedrich 1992, pp. 140–142; Lamping 2008, p. 13). One researcher, therefore, argues that the period around 1900 was characterized by a “homogeneity of the heterogeneous” (Fähnders 1998, p. 11). Despite these limitations, most scholars would probably agree that modernist poetry is at least more heterogeneous than the poetry of realism.

Hypothesis 3: There is a fundamental ‘rupture’ between modernist poetry and earlier, more traditional poetry. This view was already held by contemporary authors, critics, and anthologists, who spoke of a ‘revolution’ in poetry (as an example from the corpus anthologies see Bethge 1905, pp. 13–14; cf. on contemporary statements H. Friedrich 1992, p. 141; Anz 2007, p. 333; Lamping 2012; Wieland 2019, p. 17). Many researchers also emphasize major differences between modernism and previous literary periods, often using the metaphor of ‘rupture’ (H. Friedrich 1992, p. 20; Kiesel 2004, pp. 141–142; Frick 2007, pp. 97–98; Goltschnigg 2007, p. 169; Lamping 2012; Andreotti 2014, p. 5; without this metaphor: Klinger 2002, p. 160; Lamping 2000, p. 140; Lamping 2008, pp. 11, 13). But the ‘rupture’-thesis is also partly qualified. For example, it is emphasized that modernism still refers to traditions (even though it uses them in new ways) (Kiesel 2004, pp. 142–143; Frick 2007, pp. 98 –99; Goltschnigg 2007, p. 169). Others argue that many relevant authors were located somewhere between realism and modernism or that they combined traditional as well as new elements, which implies a smoother transition between periods (see for C. F. Meyer Selbmann 1999, pp. 149, 152; for Fontane (!) Selbmann 2007, p. 201; for Baudelaire, Rilke, Hofmannsthal, and Kafka Lamping 2012). Still, others relativize the novelty of modernism in general (Hiebel 2005, p. 27; Anz 2007, p. 333). Thus, hypothesis 3 is partly controversial in research.

It is possible to combine the aforementioned hypotheses in a visual model. The purpose of this model is threefold: it visually summarizes the research hypotheses, it relates the hypotheses to one another, and it demonstrates that all hypotheses about similarity and dissimilarity combined offer a fairly comprehensive interpretation of the transformation from realism to modernism, again underscoring the relevance of similarity as a category of analysis.

In the model, each point represents a poem. The greater the distances between the points, the more dissimilar the texts. The distances are not based on calculations but on a hermeneutic understanding of the research and are meant as rough approximations of general ideas. One can see that the distances within realism are smaller than the distances within modernism. It is also visible that there is a strong division between realism and modernism and that the naturalist poems tend to gravitate more towards realism than modernism.14

Admittedly, this model is not explicitly advocated in research. Only rarely does a single scholar state all the hypotheses that the model synthesizes. Like any model, it represents only a section of reality and neglects other aspects, such as the differentiation of individual dimensions of similarity, or synchronic and diachronic period-internal differentiations of, for example, individual authors, groups of authors, or literary movements. Some aspects of the model are, as explained, controversial in research, but it is all the more interesting to examine whether our results fit the model and the underlying hypotheses.

For a first exploration of the (dis)similarities between realism and modernism, we project the poems into a two-dimensional space (Figure 6). Some similarities with the model derived from research (Figure 5) become apparent. In particular, a distinction between realism and modernism is evident, even though the separation is far from perfect since there are numerous overlaps between the two periods. Furthermore, it is consistent with the research model that the naturalist poems tend to stay within the realist spectrum and hardly enter ‘decidedly modernist’ areas. However, it is necessary to test the hypotheses from literary studies more precisely than just by explorative means.

Hypothesis 1: The poetry of naturalism, as represented in the anthology Moderne Dichter-Charaktere, is predominantly traditional rather than modernist To test the first hypothesis, we examined the similarity of the programmatically naturalistic anthology Moderne Dichter-Charaktere to the realism and modernism corpora. In addition, we measured the distances between the poems within the latter two corpora to be able to assess the comparative analyses more accurately.

The boxplots show that overall and for each individual dimension ‘content’, ‘form’, ‘style’, and ‘emotion’ the distances between naturalism and realism are smaller than the distances15 between naturalism and modernism. At the same time, the distance between the naturalism and realism corpus is larger than the distance between the poems within the realism corpus. Surprisingly, in the dimension ‘content’ no higher proximity to the modernism corpus is seen.

A stronger similarity between naturalist and modernist poems would have been expected based on the literary-historical theses we have mentioned above. As expected, the analyses support the thesis that the naturalism corpus is more similar to the realism corpus than to the modernist corpus. However, a more detailed look shows that differences can be found in the individual dimensions. This could indicate that the naturalistic poems probably do not use the same means as realistic poems. What exactly these differences are should be investigated in a further study. However, equating naturalist with realist poetry falls short in any case since the internal distance in the realism corpus is smaller than that in the comparison between naturalism and realism. It should be emphasized that we have studied the effect only for the anthology Moderne Dichter-Charaktere and only using its short poems, as stated above. Further study would have to take into account that the modernism corpus also contains some naturalistic poems.

Hypothesis 2: Modernist poetry is heterogeneous, that is, more heterogeneous than realist poetry. From now on, when we compare realism with modernism, we no longer include the naturalist poems in our calculations and visualizations, since we have seen that naturalism is located somewhere between realism and modernism. However, we now distinguish in modernist texts canonical and non-canonical authors in order to point out some peculiarities of the canonical poems.16

To test hypothesis 2, we compare the distances within realism with those within modernism (Figure 8). In all dimensions, the distances within modernism are greater than in realism, most clearly in the dimension ‘form’. Thus, the hypothesis that modernist poetry is more heterogeneous than realist poetry can be confirmed by our data. However, the differences in heterogeneity are mostly small and should not be overemphasized. Modernist poems by canonical authors are slightly more heterogeneous than non-canonical poems regarding the dimension ‘form’. Otherwise, the canonical poems are not characterized by greater distances among themselves than non-canonical modernist poems. On the contrary, the distances for the dimensions of style and especially emotion are much smaller within the canonical texts than within the non-canonical modernist poems. All in all, the canonical texts are no more heterogeneous than the non-canonical ones. This is surprising, since one might have expected a particularly high degree of individuality and thus heterogeneity in the canon. In any case, it must be kept in mind that the subcorpus of canonical modernist poems is very small (58 poems, 5 authors), which limits the validity of the results. Further research is needed here.

Hypothesis 3: There is a fundamental ‘rupture’ between modernist poetry and earlier, more traditional poetry. Some difficulties arise in testing hypothesis 3. It is not clear what exactly is meant by ‘rupture’ and how we should measure it. One possibility is to assume that the term ‘rupture’ in hypothesis 3 denotes a certain kind of literary change, namely a change that (a) is particularly large compared to other changes between periods (e.g. between romanticism and realism) and that (b) occurs abruptly, i.e. in a very short period of time. However, we do not have data on other changes between literary periods and we cannot analyse whether the shift between realism and modernism was a continuous, decades-long process or a matter of a few years.

So while we are not able to test hypothesis 3 directly, we can at least share some observations that are likely to be related. In particular, we will compare the distances between realism and modernism with distances within realism. If the distances between realism and modernism are greater than within realism, it can at least be said that modernism is different from realism.

In all dimensions, the distances between realism and modernism are larger than the distances within realism. However, the differences are not enormous. Moreover, the two-dimensional plot above (Figure 6) shows that modernist poems appear not only outside realism, but often within the realist spectrum as well. While these observations cannot falsify hypothesis 3 directly, they certainly do not confirm the hypothesis either. If anything, our results suggest that the notion of a fundamental ‘rupture’ between realism and modernism might be exaggerated.

One could assume that researchers use the metaphor of ‘rupture’ because they focus on other, namely canonical texts. The distance between realism and canonical modernism is indeed larger than the distance between realism and non-canonical modernism regarding the form, and at least a tiny bit larger for the dimensions ‘content’ and ‘overall’. But in terms of style, canonical modernism is no further from realism than non-canonical modernism, and in regards to emotion, the distance between realism and canonical modernism is even smaller than between realism and non-canonical modernism. Thus, our results do not show that the distances between canonical modernism and realism are systematically larger than between non-canonical modernism and realism. The idea that the canonical texts set a trend that the non-canonical texts follow, just not as decisively, cannot be confirmed.

One might expect the canonical modernist poems to be at least closer to the non-canonical modernist texts than to the realist poems, but this is not true either, according to our data: The distances from canonical modernist poems to realist texts on the one hand and to non-canonical modernist texts on the other hand do not differ significantly. In the case of the dimension ‘form’, the canonical modernist poems are even closer to the realist ones than to the non-canonical modernist ones.

The results for the canon are counter-intuitive and call for further research. Again, our observations may have something to do with the fact that our subcorpus of canonical texts is very small and that we only analyze short poems.

To further explore the differences between modernist and realist poetry in our vector space, we constructed a timeline from a graph network. The network was created using all pairwise distances (or similarities more precisely) between the document vectors. For all dimensions except ‘form’, the distances are based on the vectors of paraphrase-XLM-R, after the adaptation with similarity learning. For ‘form’, only the formal feature vector similarities were used. All distances were standardized per dimension to lie between 0 and 1 (due to the different metrics used to determine the vector distances).

Each node in the graphs represents a span of 5 years (i.e. 1865 for the span 1863-1867). The edge between two year slices is depicted by the mean distances of a sample of 30 poems – if less than 30 poems were available, poems were drawn multiple times. The alpha of one edge between two years visualizes the degree of their similarity based on the chosen poems. We only used poems where the corresponding years were manually checked and corrected by us if necessary. This amounted to 321 poems between 1845 and 1911 specifically.

From this visualization which is based not on the assignment of the poems to a period by the editors of the anthologies, but on the publication date of the poems, we can make some observations. In terms of form, we can surmise from the timeline that realist poems are more similar to each other and thus more homogenous than modernist poems are (coinciding with our findings from hypothesis 2). Additionally, the further the nodes are away from realism, the weaker the similarity becomes, implying that later modernist poems become even more estranged from the form of realist poems. The networks for content and style seem similar: both suggest a kind of split between the epochs, hinting at the possibility that modernist and realist poetry have a higher inter- than extra-epochal similarity (coinciding with our findings from hypothesis 3). The timelines could potentially not only help with identifying whether a rupture between the epochs exists or not but also when exactly such a rupture occurs. While ‘style’ shows its split around 1880, the split for ‘content’ appears to be at around 1885, implying that the change from realism to modernism first became apparent in style and then in content. For ‘emotion’, we cannot discover any kind of pattern in the timeline, suggesting that emotions thematized or expressed in the poems might contribute to a continuity between the two epochs.

In summary, we were able to confirm some important hypotheses from literary studies, while differentiating or relativizing others. Our data supports the view that naturalist poetry is closer to realism than to modernism; however, simply equating naturalist and realist poetry would not be appropriate. We showed that modernist poetry is indeed more heterogeneous than realist poetry, even though the differences are limited. Finally, our findings suggest that the change from realism to modernism was an evolutionary transition rather than a revolutionary disruption. The results encourage increased attention in literary history to processes of gradual, limited change, rather than thinking only in terms of either stasis or rupture.

The assumptions made in this section are still only based on exploratory visualizations and comparatively little data. Subsequent research could expand this subcorpus of year-annotated poems (most importantly including longer poems as already mentioned) while further research questions could investigate these assumptions, e.g. whether the rupture between the epochs could have happened at slightly different points in time for different dimensions or whether ‘form’ really is the most suitable dimension to measure homogeneity and heterogeneity within realism and modernism for example.

In a recent article (Underwood and So 2021) discuss the question of whether the mapping of cultural artifacts to some spatial representation is not ‘distorting’ them, whether cultural relationships obey a spatial logic at all. Their experiments show that even if we have some seemingly convincing arguments against this kind of mapping, we accumulate more and more empirical evidence that it works very often astonishingly well. Our paper adds to this evidence: Textual representations in high-dimensional space seem well-suited to express even complex text models though more empirical work may expose its shortcomings in the future. In the meantime, we hope our approach can be used to reevaluate our understanding of the fundamental concept of similarity, not only in Computational Literary Studies.