BookNLP-fr, the French Versant of BookNLP. A Tailored Pipeline for 19th and 20th Century French Literature

2.1 Computational Methods Applied to Literary Text Analysis

Statistical methods have been used extensively in literary text analysis to identify patterns and trends in large amounts of textual data. Different pieces of software are available for this, for example, quanteda (Benoit et al. 2018), stylo (Eder et al. 2016), TidyText (Silge and Robinson 2017), or Voyant Tools (Rockwell and Sinclair 2016), to cite the most famous. They are available ‘off the shelf’, which means that they can be used directly by scholars and researchers to analyze texts. These tools can handle raw text directly, or after basic NLP processes such as lemmatization, part-of-speech tagging, or other kinds of annotation. They offer various visualizations to interpret the texts, such as dendrograms to represent the ‘distance’ between various books in a corpus, or charts to show what type of vocabulary is typical to one author as opposed to another.

There are clear benefits in using statistical methods to analyze literary texts, such as the ability to process and analyze large amounts of data quickly and efficiently, to identify patterns and trends that might not be apparent through traditional close reading methods, and to generate new research questions and hypotheses. NLP is needed to better represent the content of the text, i.e., what the text says behind the words used. NLP techniques can be used to annotate literary texts by providing syntactic and semantic annotations. NLP has become an increasingly important tool in the field of literary studies, providing new methods for analyzing and interpreting literary texts. NLP tools (e.g., NLTK (Bird et al. 2019) or Stanford CoreNLP Toolkit (Manning et al. 2014)) have been used to perform a wide range of tasks, including part-of-speech tagging, syntactic analysis, named entity recognition, etc. In the following paragraphs, we will specify the linguistic analyses available by the BookNLP pipeline: entity recognition, coreference resolution, event recognition and quotation detection. The tools mentioned in the paragraph above do not provide these types of semantic analyses, but only use morphological and grammatical linguistic analyses. BookNLP thus occupies a special niche and provides more semantically oriented annotations.

Entity Recognition. Entity recognition, along with coreference resolution, is of prominent importance, since it makes it possible to track characters, their actions, and their relationships over time. Named entity recognition (NER) is a well-established task in NLP, referring to the recognition of persons, locations, companies, other institutions, etc. (Maynard et al. 2017). NER systems exist for a wide array of languages (Emelyanov and Artemova 2019), with generally good performance, depending of course on the nature of the document to be analyzed and of the gap between training data and target data. Recognizing mentions referring to characters in a novel shares many features with NER, but is more varied (not all characters have a name, and a character can correspond to an animal, for example). Locations are also of the utmost importance to track the movements of characters (Ryan et al. 2016), but also to detect events. Note that performance may vary greatly depending on the nature of the text and of the entities to be recognized, for example, in the novel Les Mystères de Paris, written between 1842 and 1843 by Eugène Sue, most of the characters have names that are similar to noun phrases, such as “la Goualeuse” (meaning the Street Singer) or “le Chourineur” (meaning the Stabber). Also science fiction, which is full of non-classical proper nouns, can be very challenging for the task (Dekker et al. 2019). A module able to predict, or at least estimate, performance from cues gathered in the text would be useful to process large collections of novels.

Coreference Resolution. In the sense of linking together all the mentions in the text of a given character, although the task can involve all kinds of names or even nouns, coreference resolution is challenging by nature. There is a long tradition of research in coreference resolution in NLP, and modules exist for different languages, with various levels of performance (Poesio et al. 2023). The quality of the different systems is still increasing (through end-to-end models (Lee et al. 2017) and then transformer-based language models (Joshi et al. 2019)), and coreference resolution remains a very active field of research in NLP. The task is more challenging for French or Russian than for English, since the ‘it’ pronoun limits ambiguity in English, whereas in French all nouns are masculine or feminine, not only human beings, and are referred to with third-person pronouns. For instance, in “Marie veut qu’on lave la voiture, elle est sale.” (“Marie wants that we wash the car, it is dirty.”), elle refers to the car but could theoretically also refer to Marie; from a human point of view there is no ambiguity in this sentence, but the analysis requires semantic information. When applied in literary studies, automatic coreference systems often break long coreference chains due to the fact that they use a fixed-sized sliding window. If a given character does not appear during a certain period of time (i.e., a certain number of pages), it makes it harder to retrieve its antecedent. Literature provides a good testbed for the coreference task, since novels are long, real, and complex texts on which performance can (and should) still improve a lot.

Event Recognition. Event recognition involves the automated identification and extraction of verbs and, more rarely, nouns referring to events. The task is difficult in that there is no clear definition of what an event is, and other features interact with the definition (among others: negation, adverbials, and modals), and not all occurrences of verbs should be annotated (e.g., in “I like to play tennis”, play is an infinitive that refers to something I like, but it is generally considered that there is no event per se in the sentence). As for literary texts, there have been initiatives to annotate events (Sims et al. 2019), but most verbs and even some nouns can refer to events (Hogenboom et al. 2016; Sprugnoli and Tonelli 2016), which may lead to a too fine-grained annotation. There is thus a need to redefine the task and provide an intermediate level of annotation, between isolated events and the novel as a whole (Lotman 1977; Schmid 2010a, b), but also higher level annotations (like the notion of scene) have proven difficult to formalize, leading to very low accuracy in practical experiments (Zehe et al. 2021).

Quotation recognition plays a crucial role in enhancing the understanding of textual content by identifying and isolating direct speech instances. This feature is instrumental in extracting and preserving the spoken words of characters, enabling a fine-grained analysis of dialogue patterns and character interactions (Durandard et al. 2023; van Cranenburgh and van den Berg 2023). A crucial but complex part of the task consists in establishing which character is at the origin of a given utterance. A recent study has shown that performance on this task is still rather low and would need to be improve to be really usable in operational contexts (Vishnubhotla et al. 2023).

2.2 The BookNLP Project

BookNLP is a set of NLP modules designed specifically for the analysis of novels and other literary prose texts. Developed by David Bamman and colleagues at the University of Berkeley (Bamman 2021; Bamman et al. 2014), BookNLP employs a combination of machine learning and linguistic analysis techniques to extract information from text and perform tasks such as character recognition, coreference resolution, event recognition, and quotation extraction.¹ The annotated files that are available for training constitute the LitBank corpus (Bamman et al. 2019, 2020). This corpus is publicly available², which makes it possible to regularly retrain the system as NLP continues to evolve rapidly (especially LLMs).

Entity Recognition. One of the primary tasks of BookNLP is entity recognition, more specifically the recognition of characters, locations, and vehicles, showing the focus on the actions of characters. This information is used, i.a., to study how mobile protagonists are and what kind of space male and female characters occupy (Soni et al. 2023). Character recognition is often coupled with other information (such as gender, attributes, relations between characters) that can be useful for sub-stream tasks.

Coreference Resolution. In the context of literature, coreference resolution often involves resolving pronouns and other referring expressions to specific characters or entities. BookNLP employs advanced linguistic analysis to identify and link references to the same entity, and the extra knowledge provided by LLMs is especially useful for this task.

Event Recognition. Another essential task performed by BookNLP is Event recognition. It should be crucial for analyzing the development of the storyline and identifying key plot points, but the huge number of verbs supporting actions makes the annotation too prolific and not adapted to specific needs. The proper annotation of negations, adverbs, and modals is also an open problem. This is why event recognition has not been addressed as a priority in the context of the Multilingual BookNLP Project, which rather focus on entity recognition and coreference resolution.

Quotation Extraction. BookNLP is equipped with the capability to extract quotations from a text. This involves identifying and isolating the direct speech or quoted passages within the literary work. Accurate quotation extraction is vital for understanding character dialogue, the intentions of characters, and developing further analyses. However, quotation recognition without speaker attribution is not so useful, and as we have seen, speaker attribution remains an open question, as accuracy for the task remains low (Vishnubhotla et al. 2023).

The application of BookNLP for the analysis of novels and other literary works aims at providing a deeper understanding of narrative structures, character dynamics, and thematic elements in novels (Piper et al. 2021). The different modules are intended to assist researchers in literary analysis, but also in digital humanities and cultural analytics.

3.1 The Training Corpus and the Democrat Project

The Democrat Project (hereafter just Democrat), led by Frédéric Landragin (2016, 2021), funded by the French National Research Agency (ANR) and completed in 2020, aimed to develop an annotated corpus at the level of coreference chains in French. Before the Democrat, no such corpus existed. One of the fundamental aspects of Democrat was the annotation of long texts, in contrast to, e.g., the Ontonotes corpus (Weischedel et al. 2013), which serves as a standard for English but is predominantly composed of short texts. Additionally, the Democrat project aimed to annotate a wide variety of text types, including novel chapters, short stories, journalistic pieces, legal documents, encyclopedic entries, technical texts, and more. It also had a diachronic dimension, spanning from medieval French to contemporary French.

For the needs of the BookNLP-fr project, we focused on annotations related to novels and selected texts spanning from the early 19^th century to the early 20^th century. Before this period, French is more prone to variation, and for the more recent period, texts are not freely shareable due to copyright issues. Lastly, to keep the annotation task manageable, each text in the Democrat corpus is actually composed of a 10,000-word excerpt (leaving us with 184,137 tokens). In addition to this selection from Democrat, we added two short stories by Balzac, which account for 45,238 tokens. Information about these texts and those from Democrat can be found in Table 1.

3.2 Data Preparation and Annotation

In the scope of the Democrat project, annotations have been applied to all types of coreference. However, for the BookNLP-fr project, our specific focus lies within a subset of these coreferences, corresponding to certain types of entities: persons (PER), facilities (FAC), geo-political entities (GPE), locations (LOC), vehicles (VEH), organizations (ORG), and denotations of time (TIME). Definitions from all these categories except for time are adapted from Bamman et al. (2019).

PER. According to Bamman et al. (2019, 2139): “By person we describe a single person indicated by a proper name (Tom Saywer) or common entity (the boy); or set of people, such as her daughters and the Ashburnhams”. Examples of PER from our corpus can be found in example (1) and example (2)³:

FAC. We follow Bamman et al. (2019, 2139)’s definition: “For our purposes, a facility is defined as a ‘functional, primarily man-made structure’ designed for human habitation (buildings, museums), storage (barns, parking garages), transportation infrastructure (streets, highways), and maintained outdoor spaces (gardens). We treat rooms and closets within a house as the smallest possible facility”. See example (3):

GPE. We follow Bamman et al. (2019, 2139)’s guidelines for this category: “Geo-political entities are single units that contain a population, government, physical location, and political boundaries.”. See example (4):

LOC. As opposed to GPEs, Bamman et al. (2019, 2139) define locations as “entities with physicality but without political organization […] such as the sea, the river, the country, the valley, the woods, and the forest”. See the example (5) and the example (6) from our corpus:

VEH: The definition for a vehicle is a “physical device primarily designed to move an object from one location to another” (Bamman et al. 2019). An example from our corpus:

ORG. According to Bamman et al. (2019, 2139), “[o]rganizations are defined by the criterion of formal association”, such as the church or the army. An example from our corpus can be found in example (8):

TIME. This category is absent in the annotations of Bamman et al. (2019). We designed it to annotate temporal information, duration indications, and moments of the day (e.g., day, night, morning). See the example (9) and the example (10) from our corpus:

As part of the refinement process, the initial annotations required a thorough revision and cleaning. We had multiple team discussions about many borderline cases, such as whether gods and Greek heroes should be annotated as characters, the status of speaking animals, and the exact distinction between GPE, FAC, and LOC. We meticulously documented every choice made during the annotation process. This documentation is publicly available in an annotation guide⁴ and provides a valuable resource for understanding our decisions and methods in characterizing entities in the context of the BookNLP project, based on the initial groundwork provided by the Democrat project. Once the annotation guidelines were finished, the entire corpus was annotated by freshly trained annotators. Their first annotations (comprising 315 tags produced during their training phase) featured an inter-annotator agreement score of 0.38 Cohen’s kappa, meaning a fair and almost moderate agreement (Cohen 1960), but showing that this is not a trivial task. With better-trained annotators, values between 0.70 and 0.75 were reached, providing a reasonable basis for further training models. Most of the errors were due to forgotten mentions, and uncertainties about difficult cases (plurals, fuzzy expressions, non-referential entities). Another look at the annotated files by another trained annotator makes a huge difference to get a better and more homogeneous coverage (especially concerning forgotten entities during the initial annotation stage).

After annotation, to facilitate seamless integration with the BookNLP software, the annotations were transformed into a compatible format. We annotated the entity types in TXM (Heiden 2010) because the Democrat corpus is distributed in this format, and later migrated our annotations to brat (Stenetorp et al. 2012), the format used by the team in Berkely. The number of entities in each category can be found in Table 2.

3.3 Software Development

LLMs play a prominent role in contemporary NLP. Our implementation of BookNLP-fr is built upon the software from the Multilingual BookNLP Project. Two separate models are developed for the two tasks we perform (namely entity recognition and coreference resolution). Entity recognition is performed before coreference resolution.

Detecting the literary entities, a BiLSTM-CRF model (Bamman et al. 2020; Ju et al. 2018) is fed with contextual embeddings from the CamemBERT model (Martin et al. 2020), which is a BERT-based architecture (Devlin et al. 2019) tailored for French.

For the coreference part, a BiLSTM is also fed with the embeddings from CamemBERT. Following Bamman et al. (2020), who in turn follows Lee et al. (2017), the BiLSTM architecture is attached to a feedforward network in which the probability that two mentions (detected entities) are coreferent with each other is evaluated. Mentions are linked to their highest scoring antecedent (a null-antecedent is always an option), and coreference chains are defined as the transitive closures of links.

For each model, we split the corpus into training (80%), development (10%), and test (10%) sets. The results can be found in section 4.

While event annotation remains a focal point, challenges persist, primarily due to limitations in performance and the inherently ambiguous nature of defining events. The elusive nature of the concept makes it challenging to generate consistently relevant and usable results. As for quotation identification, we acknowledge the need to integrate speaker recognition for a more comprehensive understanding of textual nuances.

Given these considerations, we have more specifically directed our efforts toward optimizing modules for entity recognition and coreference resolution. This focus allows us to refine and train models that are specifically accurate in identifying and linking entities within a given text, contributing to the effectiveness of BookNLP-fr for downstream tasks (like subgenre classification, see section 5).

4.1 Named Entity Recognition Evaluation

Table 3 reports our results for entity recognition, traditionally measured through precision (the percentage of entities correctly recognized among those recognized) and recall (the percentage of entities correctly recognized among those to be recognized). Please note that ORG is absent from this evaluation, because due to an uneven distribution of this tag in different texts, it was only present seven times in the test corpus, making the estimation of precision and recall unreliable.

When assessing the model’s performance, a higher precision relative to recall suggests that the model is more likely to make accurate predictions when identifying literary entities. Precision denotes the percentage of correctly predicted literary entities among all entities predicted by the model. High precision is advantageous, ensuring that the identified literary entities are more likely to be accurate, albeit at the potential cost of missing some relevant entities (lower recall). Prioritizing precision in this context aids in minimizing false positives, thereby enhancing the reliability of the identified literary entities. It is important to highlight that literary entities differ from typical named entities in NLP, displaying a much larger range of possibilities. Consequently, the obtained results, though seemingly divergent from NLP standards, represent a pioneering achievement in the analysis of French fiction, as this is the first study of its kind. Some scores may appear modest compared to the state-of-the-art, particularly regarding the recall for TIME expressions. This is due to the extensive diversity of time expressions in our corpus, which is far more varied than in the traditional news corpora typically used in NLP, coupled with the limited number of examples in the training corpus (see Table 4 for a comparison with a state-of-the-art system). Nevertheless, we have opted to report these scores for the sake of comprehensiveness. In the near future, we will strive to expand the coverage of our system, aiming to achieve improved recall across various categories beyond PER.

As a baseline, we ran the CamemBERT-NER model⁵, which is a NER model fine-tuned from CamemBERT on the wikiner-fr dataset. Table 4 shows the baseline performance compared to BookNLP-fr. The results show that BookNLP-fr is as good as the fine-tuned model for proper name recognition, but it captures much more by including pronouns and common nouns, which the baseline model does not handle at all. The F₁ score for the detection of PROP/NOM/PRON mentions reaches 83.13, which is in line with the English BookNLP (88.3). BookNLP-fr thus demonstrates its robustness for the classic task of proper name recognition. However, the real value of our model lies in its ability to go beyond this to capture the full spectrum of what constitutes a character in novels. This aligns with Woloch (2003)’s concept of the character space as “the encounter between an individual human personality and a determined space and position within the narrative as a whole”, allowing automatic detection and analysis of the distribution of character mentions throughout the narrative (Barré et al. 2023).

4.2 Coreference Resolution Evaluation

Table 5 presents the evaluation metrics for coreference resolution using BookNLP-fr on our test corpus. Three key metrics, namely MUC, B³, and CEAF_e, are employed to assess its performance. As coreference chains are complex to model, different evaluation metrics are necessary to get a global image of the system’s performance. We refer to Luo and Pradhan (2016) for a comprehensible explanation of these metrics. Our average F₁ score is 76.4, calculated as the mean of the three metrics. The reported scores suggest a commendable performance, but the practical utility in the context of literary analysis should be further explored based on the specific goals of the research or application. Note that the English BookNLP yields 79.3 in performance for the same task.

The challenge of duplication arises when the model detects the same character multiple times within the analyzed text. In some instances, among the top five literary entities identified by the model, there may be cases where two or more main characters share the same name or attributes. While this duplication might initially raise concerns, e.g. in the study of character networks (Perri et al. 2022) or the overall number of characters in novels, it may not pose a significant issue when the focus is on character characterization. For example, in studies of the representation of male and female characters, the output of BookNLP has proven useful (e.g., Gong et al. 2022; Hudspeth et al. 2024; Naguib et al. 2022; Toro Isaza et al. 2023; Underwood et al. 2018; van Zundert et al. 2023; Vianne et al. 2023).

Also in the following case study, the primary objective is not to pinpoint unique and distinct characters, but rather to establish a proxy for characterization as a whole. Our goal is to capture the prevalence and significance of certain characters across various texts and literary works. Hence, the emphasis lies more on character representation and the overall impact of these characters on the literary landscape, rather than on identifying entirely separate and non-repeating characters.

5.1 Introduction

This case study aims to demonstrate that BookNLP-fr can be of significant assistance in the realm of computational literary studies (CLS). We illustrate this assertion through a canonical issue in CLS: the automatic detection of literary genres. Historically, the division of novels into specific subgenres has been a classification practice employed by literary stakeholders such as librarians, editors, and critics. This practice is partly justified by a specific textual component that relates to the spatiotemporal framework, characters, themes, or narrative progression.

Genre is a central concept in poetics, successively defined by theorists from Aristotle to the structuralists through the romantics and Russian formalists (i.a. Aristote 1990; Bachtin [1987] 2006; Genette 1986; Schlegel et al. 1996). From our computational standpoint, structuralists have offered intriguing definitions. For example, Schaeffer (1989, 73) defines genericity as an “internalized norm that motivates the transition from a class of texts to an individual text conforming to certain traits of that class”. There could be a set of textual procedures internal to works, and the mission of CLS would be to find the best ways to account for that fact. However, the norms or formal rules of subgenres cannot be solely boiled down to formal or thematic rules. The sociological approach, as illustrated by Bourdieu (1979), emphasizes the influence of reader communities in defining genres, examining power dynamics and the accompanying aesthetic hierarchies in the literary field. Nevertheless, these norms do indeed exist, as they enable a work to conform to the established and shared use of a “horizon of expectations” (Jauß 1982, 22) audience, which might induce authors to adhere to certain expected norms and styles.

Various studies have devised strategies to automatically identify subgenres. Selected studies have employed methods such as the bag-of-words (BoW) (Hettinger et al. 2016; Underwood 2019) or topic modeling (Schöch 2017; van Zundert et al. 2022) to find subgenre similarities between texts. In addition to these basic features, researchers utilize machine learning techniques in a supervised setting, employing methods such as logistic regression or support vector machines when ground truth is available. However, the challenge often arises from the potential incompleteness or temporal bias of these ground truths. Unsupervised learning approaches and clustering methods have also enabled the exploration of hybrid texts that belong to multiple subgenres, as demonstrated by studies like (Calvo Tello 2021; Sobchuk and Šeļa 2024). In our case study, we will rely on a corpus with predefined labels, while acknowledging the idea that subgenres are not monolithic categories. Thus, the objective is not so much to demonstrate the validity of subgenre labels, which are often incomplete or limiting in reality, but rather to show that the interpretability of errors in automatic classification can lead us to a more nuanced and comprehensive understanding of the subgenre phenomenon.

Despite recent advancements in NLP, the bag-of-words approach remains largely unchanged. This is because many tools, including document embeddings, are not easily interpretable and are optimized for short texts. In this context, we present in the next section a method that aims to find a balance between the use of state-of-the-art methods for literary text processing and their interpretability.

5.2 Method

5.2.1 Corpus and Subgenre Labels

Our case study is built on one of the largest corpora for fiction in French: the corpus Chapitres, a corpus of nearly 3,000 French novels (Leblond 2022). The investigated period covers over two centuries of novel production, from the 19^th to the 20^th century, as can be seen in Figure 1.

Figure 1: Distribution of the number of tokens over time.

Approximately two-thirds of Chapitres is annotated with subgenre labels. This annotation is based on the classification of the French National Library (BNF). We choose to concentrate our analysis on the five most prevalent subgenres within the corpus: adventure novels, romance, detective fiction, youth literature, and memoirs. The validity of these labels is not clearly established, as the practices of the BNF for assigning these labels have not been systematized nor standardized. Therefore, there is no ground truth per se, but our supervised approach described in subsubsection 5.2.3 aims precisely to understand the boundaries of subgenres.

5.3 Results

5.3.1 BoW vs. BookNLP-fr Features

Table 6 and Table 7 display the classification report of the models’ evaluation on the test set. Both models achieve good results: 72% accuracy for the BoW-based model and 75% for the BookNLP-fr-based model. This means that our models are capable to correctly identify the subgenre three out of four times, whereas a random baseline yields an accuracy score of 0.2. The main result here is that differences exist among our subgenres, whether from the perspective of text structure with MFW or from a semantic standpoint with BookNLP-fr. The fact that the BookNLP-fr-based model obtains an additional 3 points of accuracy might not be revolutionary, but the primary argument for this type of feature extraction lies more in the interpretation of features, as discussed in subsection 5.4.

Table 6: Classification report for BoW.

	Precision	Recall	F1	Support	Accuracy
Children	0.75	0.75	0.75	130
Memoirs	0.79	0.82	0.80	130
Detective	0.67	0.68	0.67	130
Adventure	0.60	0.65	0.62	130
Romance	0.84	0.72	0.80	130
Full Dataset				650	0.72

Table 7: Classification report for BookNLP-fr features.

	Precision	Recall	F1	Support	Accuracy
Children	0.65	0.79	0.71	130
Memoirs	0.78	0.89	0.84	130
Detective	0.68	0.70	0.70	130
Adventure	0.73	0.73	0.73	130
Romance	0.90	0.65	0.75	130
Full Dataset				650	0.75

Table 8: BookNLP-fr features accuracy.

BookNLP-fr Features	Accuracy
LOC	0.45
FAC	0.59
VEH	0.42
GPE	0.47
TIME	0.50
PATIENT	0.52
AGENT	0.62
ADJ	0.50
Baseline	0.2

To enhance our comprehension of how the models behave and the nature of their errors, we visualize their confusion matrices in Figure 2 and Figure 3. The x-axis represents the predicted subgenre, while the y-axis represents the expected subgenre. A perfect classification would display a diagonal filled with 130 correct predictions for each subgenre.

Figure 2: Confusion matrix for BoW.

Figure 3: Confusion matrix for BookNLP-fr features.

We observe that both models have quite similar error patterns, and one distinct scenario stands out: Both models predict ‘Adventure’ instead of ‘Detective’ (23 errors for BoW, 21 for BookNLP-fr). These common errors are quite understandable, since these two subgenres share many similarities, including a penchant for suspense and violent action, which could confuse the models.

Another scenario seemed highly instructive for analysis: The errors made by the models when predicting the label ‘Children’, but the expected subgenre is ‘Romance’. The BoW model performs quite well with 8 errors, but the BookNLP-fr-based model makes 26 errors. Thus, the semantic model faces more challenges in distinguishing between these two subgenres, which makes sense as both subgenres are characterized by themes centered around emotions and relationships between characters, which are common features of both subgenres.

5.4 Interpretability

This section explores the interpretation of the two SVM models (BoW-based and BookNLP-fr-based). It focuses on the misclassification of ‘Adventure’ instead of ‘Detective’.

One of the advantages of the SVM pipeline is the ability to investigate the statistical inference of the models when the kernel is in linear mode. The SVM searches for the plane in the latent space of words that best separates our two categories. Each dimension receives a coefficient, with a negative sign if the coefficient is used to predict a specific class, and a positive sign for the other. For the BoW-based model, this is quite straightforward, as a coefficient is assigned to each word (see Figure 4).

Figure 4: BoW discriminant features for ‘Adventure’ vs. ‘Detective’ classification.

Looking at the coefficients assigned for the ‘Adventure’ vs. ‘Detective’ classification, we find some relevant elements, such as the presence of the word ‘free’ (‘libre’) as the most discriminant word for assigning the ‘Adventure’ label. Apart from that, with the possible exception of the noun ‘cry’ (‘cri’), which could signify adventure, few clues remain. Verbs such as ‘dream’ (‘rêver’), ‘walk’ (‘balader’), ‘continue’ (‘continuer’), or conjunctions like ‘when’ (‘lorsque’), ‘despite’ (‘malgré’), and ‘yet’ (‘pourtant’) are not really characteristic of adventure novels. It is difficult to draw conclusions, except that these less significant coefficients seem to indicate the model’s difficulty in distinguishing between the two subgenres.

For the BookNLP-fr-based model, it is a bit more complex since the coefficients are assigned to each dimension of the Doc2Vec vectors. Therefore, we aggregated the coefficients by feature type to gain a more concrete overview of the results. Figure 5 illustrates the sum of all coefficients for each feature extracted by BookNLP-fr. We conducted a t-test to confirm that the difference between the means of the populations is statistically significant. Taking adjectives as an example (t-statistic: 28.7; p-value: 2.25 x 10^-180), we observe that the model relies more on these dimensions to assign the label ‘Detective’ compared to ‘Adventure’.

Figure 5: BookNLP-fr discriminant features for ‘Adventure’ vs. ‘Detective’ classification. ‘***’ meaning p < 0.001.

This could be explained by the strong emphasis placed on character psychology in detective novels, especially those involving criminals and detectives. For instance, in Maigret et le tueur (1969), George Simenon’s beloved detective (Maigret) is frequently characterized as ‘wise’ (‘sage’), ‘whimsical’ (‘fantaisiste’), or even ‘happy’ (‘heureux’), while criminals are ‘suspicious’ (‘suspect’) or ‘villainous’ (‘méchant’). This does not imply a lack of characterization in adventure novels but rather suggests that it is not a distinctive feature of the subgenre compared to detective novels.

Considering GPEs (t-statistic: -21.0; p-value: 8.49 x 10^-98), the reasoning is inverse: The model relies slightly more on the dimensions of the GPE vector to assign the ‘Adventure’ label than the ‘Detective’ label. This makes sense when examining GPEs for example in Les trappeurs de l’Arkansas by Gustave Aimard (1857): ‘Hermosillo’, ‘America’, ‘the New World’, ‘Guadalajara’, ‘Mexico’, etc. The novel heavily emphasizes exotic locations and mentions places in the American or Mexican West for this purpose. GPEs in detective novels are more commonplace, as these novels often take place in France, typically in an urban setting.

Thus, the model has learned that certain dimensions of characterization are more strongly associated with a particular subgenre (such as adjectives for detective novels), and that certain dimensions of the GPE or TIME vector are important for assigning the ‘Adventure’ label. In the following we generalize our approach to the entire classification process.

Examining the behavior of the coefficients when aggregated for the ten classifications, we can observe the graph shown in Figure 6. This graph depicts the model coefficients after training based on the vectors of each facet, using a dataset of 2,400 dimensions. We consider this graph as a dive into the model’s inferences, where it will assign more weight to certain categories to assign a specific subgenre.

Figure 6: BookNLP-fr discriminant features for the classification.

For example, it is observed that the value of ‘FAC’ is very high for the detective genre, indicating a particular specificity for this subgenre. Details of locations, crime scenes, investigations in specific places, detective offices, interrogation rooms, etc. are distinguishing elements of this subgenre. The same applies to ‘GPE’ for the ‘Adventure’ label, as seen previously, with an emphasis on exoticism that may play a role here, even though ‘LOC’ and ‘FAC’ do not show significant differentiation from this perspective. Conversely, for ‘Romance’ and the ‘TIME’ vector, where the coefficients for these vectors lag behind other subgenres. Examples of time in romance novels may be used more to describe emotional moments or stages in relationships rather than to highlight complex temporal events. Consequently, the model might perceive that the ‘TIME’ vector is less discriminative for this category.

We have thus demonstrated that the BoW-based classification approach is challenging to interpret, as certain highly discriminating words do not appear to bring about key distinctions between the subgenres. The BookNLP-fr-based method may offer an insightful understanding of the specificities that differentiate one subgenre from another. Both approaches do not completely substitute each other, since we are examining features of different nature (vocabulary vs. semantic), but they can complement each other to enhance interpretability.

Diving into the model’s indications, several types of features were observed to interpret the model’s inferences. Many differences among the features were noticed, although we did not have the space to interpret all of them in this article. Much work remains to be done, and new experiments should be considered, for instance going beyond the SVM, including the use of deep neural networks and textual deconvolution saliency Vanni et al. (2018), which could facilitate the return to close reading based on the embeddings derived from BookNLP-fr data.

6.1 Working with Imperfect Annotations

The utilization of computers for annotating literary texts has profoundly changed the landscape of literary studies, enabling the annotation of vast amounts of texts with unprecedented efficiency. This enables the community to address research questions that were out of reach before, such as a study at scale of characters with disabilities (Dubnicek et al. 2018), or the quantitative analysis of characters in fanfiction (Milli and Bamman 2016), and a quantitative, diachronic study of things appearing in fiction (Piper and Bagga 2022). However, this advancement is not without its challenges, particularly in the context of the inherent errors that may accompany automated annotation processes. This poses a twofold challenge for researchers engaged in the field of CLS.

Firstly, ensuring the reliability of studies based on imperfect annotations is a critical concern. Scholars must grapple with the task of guaranteeing that errors, though present, remain at a marginal level and do not compromise the validity of their research findings. This necessitates a careful balance between the benefits of computational efficiency and the maintenance of accuracy in annotations. Researchers are challenged to develop methods and quality control measures that safeguard against the potential pitfalls introduced by errors in the annotation process.

Secondly, the acceptance of computational approaches by literary scholars is not guaranteed, as the traditional paradigm within literary studies often revolves around meticulous, supposedly perfect annotations. The shift to working with non-perfect annotations, even if the errors are marginal, represents a departure from the established norm. This cultural shift within the academic community poses a psychological barrier, as literary scholars may be hesitant to fully embrace computational methods if they perceive a compromise in the level of precision to which they are accustomed.

Addressing these challenges requires not only the refinement of computational tools for annotation but also a broader cultural shift within the academic community. There is a need for transparent communication about the limitations of automated annotation processes, the establishment of best practices for mitigating errors, and the development of strategies to ensure that computational approaches align with the standards expected both in literary studies and in computer science.

6.2 Maintaining Annotation Tools in the Era of LLMs

The field of CLS is currently grappling with a significant challenge due to the rapid evolution of NLP, particularly with the proliferation of LLMs. The continuous emergence of new LLMs has led to an accelerated pace of research in the domain. While this dynamism brings about positive outcomes, such as increased research activity, the introduction of novel tasks, and the generation of new results, it also presents several inherent dangers.

One primary challenge lies in the technical aspect of keeping annotation tools up to date amidst the constant production of new LLMs by the research community and the industry. There is a delicate balance to strike, ensuring that annotation systems remain up-to-date, without expending an excessive amount of resources on incessantly adapting to the latest trends in LLM development. The challenge here is not just about technological compatibility but also about efficiently managing the resources required for frequent updates and integrations, and to produce software that is usable by a large community (i.e., software should not be dependent on an unreasonably heavy computer infrastructure).

A more critical concern revolves around the need to guarantee the reproducibility of research outcomes. The rapid evolution of LLMs implies that a specific version in use today may become obsolete or unavailable tomorrow. This raises the risk that crucial details, such as the corpus utilized, configuration parameters, and hyperparameters of the model, may not be adequately documented in research reports. Ensuring reproducibility becomes a substantial challenge as the landscape of LLMs continues to evolve, necessitating a concerted effort to establish standardized practices for reporting model specifications and associated details.

In addressing these challenges, we believe it is crucial to focus not only on technical aspects but also on developing robust frameworks for documentation and reproducibility. Establishing clear guidelines for reporting model specifications, documenting corpus details, and archiving relevant information becomes paramount for the field.