1. Introduction

For traditional literary studies approaches, the text itself is hardly ever the only subject of investigation when addressing questions related to developments in literary history. Instead, a wide range of complementary data, from letters to reviews and poetological treatises, are employed to embed a text, its production, and its reception in a broader literary historical context. Such a richness of detail and context is per definitionem not achievable when working with quantitative methods: When analyzing hundreds or thousands of texts, linking each and every one of them to their immediate context of production and reception is simply not feasible. The first hurdle of such a context-heavy quantitative approach is the lack of available data. In comparison to the entire mass of literary history, there are only few literary works which have been researched thoroughly enough to be described on all levels of production and reception. The second hurdle is that of formalization and operationalization. Even if qualitative research about all texts was available, this unstructured data would need to be digitized and operationalized to be used for quantitative analysis, again leading to a loss of detail.

Building on context-sensitive approaches suggested in previous research, it is the aim of this contribution to find an appropriate level of abstraction in “data-rich literary history” (Bode 2018, pp. 37–57) by exemplifying a workflow for the quantitative operationalization and analysis of historical literary reception, and to use this newly formalized data to investigate if external markers of reception can be predicted from features of the texts themselves. In the course of this paper, we will (1) show how to encode literary historical information in a dataset that is suitable for quantitative analysis, and apply this method to a collection of roughly 1,200 English and German novels and narratives published between 1688 and 1914 along with data on the reception of these works by their contemporaries, (2) present a nuanced and theory-based perspective on automated sentiment detection in historical literary reviews, and (3) compare contemporary experts’ reviews and a text’s popularity to textual features that reflect a text’s complexity and distinctiveness.

As part of a greater research interest in the comparative analysis of canonization processes in English and German literary history (see Brottrager et al. 2021), our approach operates between the poles of a text’s canonization status today – a result of a myriad of stacked selection processes – and its reception by its immediate contemporaries. The comparison between English and German literary history seems especially fruitful here, as their classical periods are temporally and philosophically far apart. The German classical period from 1770 to 1830 with its focus on the authorial genius and aesthetic autonomy remains a figurative yardstick for subsequent generations of writers and critics, ingraining the dichotomy of high and low literature in German literary history (Heydebrand and Winko 1996, pp. 151–157), while such a stark distinction is not encoded in English literary history. By comparing these two very different traditions over a time span that encompasses the German Classicism, but also the rise of the novel and the so-called ‘Novellenflut’ as phenomena of popular fiction, we will be able to show how initially well-received literary texts get lost in the so-called ‘Great Unread’, while others are elevated into the canon.

We will begin by discussing examples of context-rich approaches to literary reception and previous research on the categorization of reviews in the context of computational literary analyses (Section 2). This overview of practical applications will then be followed by an in-depth examination of the theoretical background of verbal judgments and evaluative actions in literary reception. Following the description of our canon-conscious corpus selection, the paper’s third and fourth section will show how historical sources of literary information can be encoded in a dataset by adding reviews as representations of verbal value judgments (Section 3) and circulating library catalogs as proxies for audiences’ interests (Section 4). The methodological part of this contribution (Section 5) will show how we have implemented a SentiArt-inspired approach (A. M. Jacobs 2019) to evaluative language for the differentiation of literary reviews. Then, we present how we used the historical data introduced in previous sections to analyze to which extent the popularity and reception of literary works can be explained with qualities of the texts themselves (Section 6). In the discussion (Section 7), we will illustrate how the theoretical framework of historical evaluation is reflected in our results.

2. Previous Work

While the examination of text-related metadata categories, such as authorial gender, genre, publication date, and broad thematic categories has already been introduced in early contributions to the field of Computational Literary Studies (CLS) (Jockers 2013; Moretti 2013), the study of reception-related data is not yet as established. Some studies have suggested measures of prestige and popularity (Algee-Hewitt et al. 2016; Porter 2018; Underwood 2019; Underwood and Sellers 2016), where these categories reflect to some degree reception-related aspects: In their publication on literary prestige, Underwood and Sellers define prestige as a dichotomy by distinguishing poems according to whether or not they were reviewed in prestigious journals (2016, pp. 323–325, see also Underwood 2019, pp. 68–110). Algee-Hewitt et al. (2016) similarly determine a text’s prestige, but do so by operationalizing the category as the number of bibliographical entries in the MLA featuring the author as the “Primary Subject Author”. Additionally, they introduce the category of popularity, modeled as the combination of the number of reprints and translations (2016, p. 3). Capturing modern readers’ responses, Porter (2018) constructs a score representing the popularity of authors by combining Goodreads metrics (the number of ratings, the number of reviews, and the author’s average rating).

The hesitation to include historical reviews as actual textual data seen in the examples above is understandable: Reviews often have to be retro-digitized before they can be analyzed, and established methods developed for categorizing shorter, more straightforward modern language reviews such as sentiment analysis are not as reliable when confronted with historical language. Du and Mellmann (2019) address these issues and suggest a multi-layered approach when dealing with historical reviews: Instead of relying solely on lexicon-based sentiment analysis,¹ they aggregated a metric that takes the distance between sentiment expression and author name into account to ensure that value judgments directly connected to an author’s work are more strongly weighted. Combined with textual features such as (lemmatized) n-grams with weights based on tf-idf and word embeddings, these sentiment values were then used to train a Support Vector Machine (SVM) which correctly identified positive, negative, and neutral sentences extracted from reviews with an overall average accuracy of 0.64 and up to 0.76 for only positive and negative sentences (Du and Mellmann 2019, p. 11).

When discussing the historical specificity of literary reviews and their implicitly marked registers (2019, p. 13), Du and Mellmann hint at elements of verbal judgments that are also extensively investigated by Heydebrand and Winko (1996) in their introductory work on evaluation in literature. According to Heydebrand and Winko (1996, p. 62), verbal value judgments can be defined as illocutionary acts of utterance through which an object is ascribed an attributive value. This attributive value in turn links back to a defined value system.² Different value systems lead to different attributive values: While in one historical context a specific characteristic is seen as valuable, it can be ascribed less value in another historical period (Heydebrand and Winko 1996, pp. 111–131, 134–162).

In addition to verbal value judgments, Heydebrand and Winko elaborate on social components of evaluation, especially those connected to selection processes. They point out that decisions for or against a text are evaluative operations that structure all levels of the literary system, from a publisher’s acceptance of a manuscript to a reader’s individual buying decision (1996, p. 79). Selective decisions by literary critics³ are especially impactful, as the existence of professional reviews spotlights a text when compared to the mass of all other published but unreviewed competitors (1996, p. 99).

Similar to our previous work on the issue of canonization (Brottrager et al. 2021), introducing an operationalization for contemporary reception based on the theoretical framework provided by Heydebrand and Winko (1996) aims at creating comparability within our own project, but is also part of a greater effort in the field of CLS to find suitable, reproducible, and adaptable implementations for complex literary concepts (see Alvarado 2019; Pichler and Reiter 2021; Schröter et al. 2021).

3. Corpora

For the compilation of our two corpora, we systematically adapted an approach proposed by Algee-Hewitt and McGurl (2015) in their contribution on creating a balanced novel corpus for the 20^th century. To tackle what they call “dilemmas of selection” (2015, p. 1), they combine existing best-of and bestseller lists with commissioned lists of novels suggested by experts of Feminist and Postcolonial Studies to create a corpus that entails multiple dimensions of canonicity: First, a very narrowly defined normative canon of the ‘best’ novels written in the 20^th century, second, financially successful and thus presumably popular novels, and third, novels belonging to an alternative canon of marginalized texts. In contrast to “samples of convenience” usually found in readily available online collections, which are “no doubt equally, if not more biased than the lists we have assembled” (2015, p. 22), using a predefined corpus list allows for a monitoring of availability issues and canonical biases.

For corpora covering the Long 18^th and 19^th Century (1688-1914), comparable lists are not or only partially available. To be able to still apply a similar logic, we had to find a way to adequately replace both existing and commissioned lists. As described above, the lists represent different dimensions of the canon, which can also be replicated when using lists of mentions extracted from differently motivated literary histories and other secondary sources. By relying on lists of texts deemed relevant by experts with different focal points, we would still be able to contrast the ”found” corpus (2015, p. 4) of already digitized material with a ”made” list (2015, p. 15) of, if not commissioned, but still purposely gathered texts. To capture the essence of normative best-of lists, we used highly condensed and consequently exclusive narrative literary histories and anthologies. Lists of popular literature and marginalized literary texts were reconstructed by including specialized sources (e.g. sources on light fiction and popular genres, companions to literature by female authors and literature from geographical peripheries) and by surveying the broader academic canon (e.g. companions to specific genres and periods).

The resulting list was then used as a basis for the corpus compilation. In a first iteration, we checked online full-text repositories.⁴ For texts not available as digitized full-text, we looked for high-quality scans or scanned and retro-digitized them ourselves. As Algee-Hewitt et al. (2016, p. 2) point out, the retro-digitization is cost- and time-intensive, which is why we did not retro-digitize all missing entries, but deliberately included texts that added a degree of diversity to our corpus because they were written by an author not already included, represent a niche genre, or other forms of marginalized literature. To ensure high-quality transcriptions, the workflow combines automated optical character recognition (OCR) and manual post-corrections.

The compilation resulted in an English corpus of 605 and a German corpus of 547 texts. The temporal distribution of publication dates in both corpora is shown in Figure 1. In both corpora, the number of texts increases around 1770, which corresponds to historically informed expectations linked to the rise of the novel in both English and German literary history. Later spikes in the English corpus are primarily caused by the inclusion of collections of (short) stories, which are incorporated as individual texts.

Figure 1

Temporal distribution of texts in our corpora.

4. Complementary Data

To be able to model literary evaluation as described by Heydebrand and Winko (1996), we expanded our dataset to include representations of verbal value judgments and readers’ selective choices. While verbal value judgments are directly preserved in historical reviews, the reconstruction of readers’ choices is not as straightforward. Transferring Heydebrand and Winko’s idea of the buying decision to the time frame in question seems impractical because particularly for earlier time periods covered by our corpora, reliable sales numbers are not available. Additionally, we wanted to introduce a measure that explicitly encapsulates a text’s popularity with lay audiences in contrast to expert opinions recorded in reviews, and historically, buying books was simply not the way the majority of readers accessed their reading materials. Here, entries in circulating library catalogs seem to be a better suited proxy: Circulating libraries relied heavily on the popularity of the items they advertised and had to adapt to audiences’ preferences in order to remain profitable (E. H. Jacobs 2003), which makes the existence of catalog entries a suitable representation of a text’s popularity.

4.1 Reviews

In both the English and German-speaking Europe, the rise of literary periodicals coincides with the commercialization of the literary market (see Italia 2012), which led to an exponential growth of available reading material and a resulting need for selection. As a consequence, literary periodicals can be seen as structuring devices (Plachta 2019) that place the reviewed texts along a gradient from well to poorly received, but also distinguish between texts that were interesting enough to be reviewed and the remaining mass of texts published at the same time. In addition to reviews being written by professional readers, numerous influential publications were directly linked to central figures of the literary sphere: Authors such as August Friedrich Kotzebue and Tobias Smollett, for example, acted as founders and editors of the Blätter für literarische Unterhaltung and The Critical Review, respectively. This direct involvement of authors as professional reviewers (see Heydebrand and Winko 1996, pp. 188–210) further accentuate the difference between evaluations by (peer) experts and popularity with broader audiences, as it is recorded in circulating library catalogs described below.

Due to the sheer number of literary journals published in the time span covered by our corpora, the selection of representative journals is based on considerations of influence and outreach, but also availability. For the English dataset, we were able to rely on some already digitized reviews accessible through the database British Fiction 1800-1829 (Garside 2011, based on Garside and Schöwerling 2000) and used the corresponding analogue bibliography for the time span from 1770-1799 (Raven and Forster 2000) to locate referenced reviews. The database and bibliography primarily list reviews in The Monthly Review (MR) (covering the years from 1800 to 1830) and The Critical Review (CR) (1800-1817), but also feature references to La Belle Assemblée (BA) (1806-1830), Flowers of Literature (FL) (1801-1809), and The Star (surveyed for 1800 through 1830). Additionally, we consulted the database The Athenaeum Project (ATH) (City University London 2001) which provides access to searchable indices of the eponymous journal published from 1828 to 1923. For the German dataset, we consulted the database Gelehrte Journale und Zeitungen der Aufklärung (GJZ18 2021), but also relied heavily on the monthly and yearly indices of selected journals which were especially influential during their respective running time: Allgemeine Literatur-Zeitung (ALZ) (1785-1849), Morgenblatt für gebildete Stände (MGS) (1807-1865), Blätter für literarische Unterhaltung (BLU) (1826-1898), and Deutsche Literaturzeitung (DL) (1880-1993).

As the available scan quality as well as the fonts and type settings differed widely across the selected publications, we trained multiple recognition models using OCR4all (Reul et al. 2019), which were then combined in several iterations of text recognition. Collective reviews of multiple texts were split into parts concerning the referenced texts, and frequently featured lengthy quotes from the reviewed texts were replaced by ellipses.

In sum, we have collected 254 English and 221 German reviews. As some of them address the same texts, this results in 197 reviewed texts in the English and 176 reviewed texts in the German corpus, which means that we were able to link almost a third of each corpus to at least one historical review. Figure 2 shows the temporal distribution of reviews for both corpora. With the exception of a major gap in reviews concerning English texts from 1820 to 1830, which is most likely caused by the running time of the surveyed journals, the reviews are quite evenly distributed from 1780 onward. The lack of data before 1780 can again be linked to the selected journals, which is why all textual analyses (see Section 6) will take this bias into account.

Figure 2

Temporal distribution of reviewed texts.

4.2 Circulating Libraries

Similar to the emergence of literary journals, the introduction of circulating libraries is closely associated with the explosion of publication numbers related to the rise of the novel and the revolution of reading in the second half of the 18^th century (Martino 1990, pp. 1–134). By lending books to people who, as Gamer puts it, “would never have considered buying fiction” (2000, p. 65), circulating libraries can be seen as a form of democratizing literary consumption. However, the libraries’ broadening target group also caused concern with contemporaries, who warned against the moral corruption caused by circulating libraries’ focus on crowd-pleasing light literature (Jäger 1982, pp. 263–264). Despite this criticism, circulating libraries became essential actors in the 19^th century literary market, with some libraries, such as Mudie’s Circulating Library, gaining so much influence and ”purchasing power” that they could single-handedly ”sell or condemn a book” (Katz 2017, p. 405).

Analogously to our approach to literary journals, the selection of specific catalogs is determined by questions of importance and availability. The issue of availability is more salient in this case: Compared to the number of preserved and recorded catalogs (Martino 1990, pp. 917–1017), only very few of them are available as digital surrogates, which limits our options quite significantly. Nonetheless, we managed to find four English and six German catalogs published between 1809 and 1907 and 1790 and 1901, respectively, allowing for an adequate coverage of the 19^th century. For the English dataset, we surveyed the 1809 catalog of W. Storry’s General Circulating Library (York), the 1829 catalog of Hookham’s Library (London), and two catalogs (1873 and 1907) for Mudie’s Select Library (London). Due to the municipal library of Vienna’s digital research focus on library catalogs, the German dataset is heavily skewed towards Viennese libraries and includes the 1790 and 1812 catalogs of rentable books at Johann Georg Binz’ bookstore, Carl Armbruster’s 1813 catalog, J. August Bachmann’s 1851 catalog, Friedrich Gerold’s 1850 catalog, and the 1901 catalog of the Literatur-Institut Ludwig und Albert Last. Linking our corpus texts with entries in these catalogs required a two-step approach: Due to the diverging formats and indexing methods and inconsistent titles and spelling variations, we combined a full-text search of automatically recognized text with a manual double-check of indices for each catalog.

Of all 1,153 novels and narratives in our corpora, 763 were referenced in at least one of the catalogs we surveyed. Especially the coverage for the English corpus is significant: 75.54 percent of all texts and 78.57 percent of all featured authors appear in at least one catalog. The same is true for 55.94 percent of all German texts and 54.34 percent of all German-speaking authors. The temporal distribution of texts available in library catalogs is presented in Figure 3. Whereas the circulating library entries for the German texts are quite evenly distributed from 1780 to 1914, there is more variance in the English corpus. From 1780 to 1890, the mean number of texts referenced in a catalog per year is 3.37, while for the years after 1890, the mean rises to 7.96. This is certainly due to the inclusion of collections of stories mentioned in Section 3, but also indicates that the last English catalog published in 1907 features many recent publications.

Figure 3

Temporal distribution of entries in circulating library catalogs.

6 Results

6.2 Regression

The highest significant correlation coefficient from the cross-validation, or the highest coefficient if none was significant, are reported in Table 10. In Figure 7, the true and the predicted scores are plotted against each other.

Figure 7

Sentiment scores and predicted scores.

Table 10

Regression results.

	English	German
SentiArt	0.113***	0.168**
TextBlob	-0.019	-0.111***
Combined	0.015	-0.126***

• ***p<0.01, **p<0.05, *p<0.1

Unlike for the classification, there is no consistent best feature combination for all regression tasks. The chunks plus copied document dataset was the best choice for English on the SentiArt and TextBlob labels, and the document plus averaged chunks dataset for the combined labels. For German, the chunks plus copied document dataset had the best performance with TextBlob and combined labels, while the document-based dataset performed best with the SentiArt labels.

7. Discussion

In all experiments, our classifiers performed better than the baselines for the three classification tasks, but only barely so for the library catalog classification with texts in German. Overall, we were able to achieve better classification results with the English than the German dataset. For both languages, predicting whether a text had been reviewed generated accuracy values that exceed the threshold of better-than-chance. This can be seen as an indication that texts that generate enough interest to receive a review share certain textual qualities. By suggesting such a relationship, the results may be seen as a consolidation of the theory presented in previous research (see Heydebrand and Winko 1996, p. 99) that the existence of a review alone – may it be positive or negative – is an important structuring device representing the attention a text attracted. The fact that a text had been added to a circulating library might be viewed as a similar indicator of interest by a broad audience. At least for the English texts, our findings indicate that there are some detectable textual qualities that can be linked to a sparked interest among the general reading public.

The higher balanced accuracy score for English library catalogs might also be connected to the distribution of data described in Section 4.2: In contrast to the German dataset, there is a clear tendency for late 19^th and early 20^th century catalogs to include contemporary texts, which can be assumed to be stylistically more homogeneous. Moreover, the last two catalogs surveyed are from the same library, Mudie’s, whose owner Charles Edward Mudie allegedly only advertised books that satisfied his personal moral and literary standards (Katz 2017, Roberts 2006). It seems plausible that these factors led to more quantitatively detectable similarities within the set of texts advertised in library catalogs and therefore to a higher balanced accuracy score.

In general, the prediction of in- or exclusions in circulating library catalogs is, with its balanced accuracy values of 0.617 and 0.539, less conclusive than the prediction of reviews. On a first glance, these differences in the classification success are surprising: The data distribution for both reviews and circulating library catalogs is similarly imbalanced; the diverging results might therefore be linked to habitual aspects of the two literary institutions. While circulating library catalogs reflect the multi-faceted and inconsistent interests of the public, the selective decisions that lead to a text being reviewed in a literary journal are more concise and uniform. In other words, the selective decisions of a smaller group of expert readers seem to be easier to predict than those based on the expected popularity with a more diverse group of lay readers.

The low correlation coefficients of the best model with 0.113 for English and 0.168 for German texts show that even with our adapted SentiArt approach, there is only a very weak correlation between the measured sentiment in reviews and textual markers of the reviewed texts. Due to this weak effect, we did not further analyze the contribution of individual features. Reasons for why combining the scores generated by the two tools leads to low and non-significant correlation coefficients could be inadequately set thresholds for switching from one tool to the other or the usage of average values for neutral reviews.

The mean validation scores across the folds of the inner cross-validations are close to the evaluation scores on the test data of the outer folds for the classification tasks, while they are a lot higher than the evaluation scores for the regression tasks. The best regression models in the inner cross-validation do not generalize well on new data and seem to overfit to the training data.

8. Conclusion

Modeling historical reception requires a dataset that encodes literary contexts by combining texts with complementary information on how they were received by their contemporaries. By operationalizing the theoretical framework suggested by Heydebrand and Winko (1996), we were able to formalize a text’s reception by experts, as well as its popularity with audiences. Differentiating these two levels of literary evaluation allows a more detailed analysis of historical reception and lays the ground work for future research on synchronic reading, diachronic canonization, and their interplay.

A conclusion of our findings is difficult due to the limited amount of available data. Historical literary data is scarce, and a larger dataset might have led to different results. Based on this data-rich literary history dataset, predicting review sentiment from texts alone proved to be successful only to a very limited extent. We had better success predicting whether literary works had been reviewed or not: There seem to be certain text qualities that make it more likely that a reviewer will pay attention and choose to review a text. To some degree, such a relationship between textual markers and selective decisions can also be detected when examining texts featured in circulating library catalogs. As circulating libraries cater to a more diverse group of readers, we assume that diverging interests represented in the catalogs could be responsible for the lower balanced accuracy values.

The low predictability of reception along with the limited predictability of a text’s canonization status, which we have shown in a previous publication (Brottrager et al. 2021), indicate that the assessment of historical novels and narratives by contemporaries as well as by a modern expert audience are mainly not due to measurable textual qualities, but the result of a complex interplay of selection and interpretation processes which are influenced by both literary and non-literary factors (Rippl and Winko 2013; Winko 2002).

Generally, our ad hoc SentiArt approach has proven to be useful for the sentiment analysis of historical reviews because it was – in comparison to the established TextBlob tool – more adept at identifying the particularities of evaluative language in reviews, as, for example, the implicitness and vagueness of negative comments. We will work on fine-tuning the word embeddings to increase the accuracy of the approach in the detection of positive reviews. Additionally, it seems reasonable to add a time component to our approach: Our corpora comprise texts from a time span of over 200 years. During this time, the market for and the status of literature changed dramatically, as did the expectations of different generations of audiences and literary experts. These historical shifts in readers’ and reviewers’ perspectives are not yet accounted for in our experiments, and we assume that all reviews express a certain sentiment in reaction to the same textual features. By extracting period-specific evaluation words and computing period-specific evaluation scores, we could account for these changes in perception, reception, and expectations.

9. Data availability

Corpora, reviews, metadata, trained word embeddings, and sentiment scores can be accessed via https://doi.org/10.6084/m9.figshare.19672410.v1.

10. Software availability

The scripts are available at https://github.com/sta-a/jcls_reception.

11. Acknowledgements

This work is part of “Relating the Unread. Network Models in Literary History”, a project supported by the German Research Foundation (DFG) through the priority programme SPP 2207 Computational Literary Studies (CLS). Special thanks to Joël Doat for his advice on the formal aspects of word embeddings.

12. Author contributions

Judith Brottrager: Formal Analysis, Data curation, Writing – original draft

Annina Stahl: Formal Analysis, Writing – original draft

Arda Arslan: Formal Analysis

Ulrik Brandes: Supervision, Writing – review & editing

Thomas Weitin: Supervision, Writing – review & editing