A Stylometric Analysis of Seneca's Disputed Plays. Authorship Verification of Octavia and Hercules Oetaeus

1. Introduction

Computational stylometry is a quantitative text analysis method mostly concerned with authorship attribution and authorship verification problems. Authorship attribution involves identifying the most likely author of a disputed document from a given set of candidates (Koppel et al. 2007, 1261), whereas authorship verification concerns the question of whether an author wrote a disputed document (Koppel et al. 2007, 1261; Juola 2015, i106). The verification task is more challenging than the attribution task because the former task involves determining whether an observed similarity in style is sufficient to verify authorship, while the attribution task merely involves picking the most similar author from the given candidates (Potha and Stamatatos 2017, 138). It is important to also note that the authorship verification typically involves both close-set and open-set scenarios. In the close-set scenario, the suspected author is one of the candidates provided, whereas in the open-set scenario, the true author may not be among the known candidates.

The main assumption behind computational stylometry is that certain words are chosen unconsciously by the writer, which form a unique, individual fingerprint of an author (Evert et al. 2017, ii4). Since these words are predominantly function words that are used in a way that is hard for the author to control, imitating someone else’s writing style is difficult for an imposter. In other words, there is an “immutable signal that authors emit involuntarily” (Päpcke et al. 2022, 1). The utility of function words in traditional and computational stylometric studies can be condensed into four points: 1) richer dataset because of their high frequency, 2) closeness of the set since function words are limited and fixed, 3) content-independent, and, as mentioned above, 4) unconscious use of them due to their high frequency (Kestemont 2014, 60; Beullens et al. 2024, 393–394).

The aim of this article is to examine whether Seneca the Younger wrote Octavia and/or Hercules Oetaeus (henceforward: Oct. and H.O., respectively) since they are both tragedies of which a plethora of literary scholars have raised concerns about their attribution to Seneca. We contribute to the debate on Seneca’s disputed texts by applying a variety of computational stylistic methods and testing several scenarios. For this we use the stylo software, an R package created and developed by Eder et al. (2016).

The ensuing sections of this study are organized as follows. Initially, a concise literature review is provided addressing Oct. and H.O. (section 2). Subsequently, section 3 outlines the rationale for selecting a specific set of imposter texts and acknowledges potential limitations associated with the limited transmission of ancient texts and differences in genre and meter. Section 4 delves into the preprocessing steps and features employed in the study, while also offering a brief explanation of each method utilized in the primary analysis. Section 5 provides a validation of the methods on texts with known authorship. Section 6 presents the main results for the disputed texts and engages in a discussion of these findings. Finally, we present our conclusions concerning the findings and outline ideas for future research (section 7).

2. Literature Review

3. Dataset

The main dataset employed in this study comprises distractor authors and verse texts that slightly precede and follow the era of Seneca the Younger (ca. 4 BCE–65 CE). In the context of computational stylometric approaches, a distractor author, or ‘imposter’, is utilized for comparison with a disputed text. For clarity, consider a text X attributed to author A, with distractor authors B, C, and D, known not to be the author of X. The soundness of a stylometric method is affirmed by observing significantly higher similarity between X and other texts by A compared to B, C, and D, confirming A as the probable true author, or vice versa.

In our analysis of Seneca, the dataset includes authors such as Ovid, Manilius, Martial, Phaedrus, Persius, Lucan, Valerius Flaccus, Statius, and Silius Italicus (see Table 10). These authors, broadly associated with the literature of the Early Empire, wrote within the first century of the Common Era (see Figure 1).⁴

Figure 1: A timeline of the authors used in the main dataset. This dataset is used for the PCA, BCT, and the first four out of five scenarios of the GI method.

In Scenario 5 presented in Table 3 (section 6), we augment the dataset used by Kestemont et al. (2016) (see Table 9 in Appendix A) with our main corpus (see Appendix A for the main corpus and Figure 2 for a visual representation of the augmented dataset used in Scenario 5 of GI). Therefore, we consider it important to explain the authors and the texts that populate this dataset, as well as their main genre. Kestemont’s dataset contains 1,850 non-overlapping text slices. The authors and the texts present in their dataset are listed in Table 9 in Appendix A. Since in their paper, they compare their corpus with Caesar’s writings, their dataset contains mostly historiographical texts and covers a huge time span (from the 4^th century B.C.E. up to the 4^th century C.E.).

Figure 2: A timeline of the authors used in Kestemont et al. (2016)’s data augmented with our main dataset. This dataset is used in scenario 5 of the GI method.

In authorship verification, the challenge of text and author selection inevitably involves some arbitrary or imperfect choices. This section aims to transparently justify our choices. According to Grieve (2007, 255), texts, whether disputed or not, are inherently tied to their historical era. Consequently, the dataset is designed to narrow the temporal scope to ensure a more focused linguistic comparison. However, we should highlight two important aspects that complicate the corpus selection.

First, besides the Senecan tragedies, there are no other extant Roman tragedies. Therefore, expanding the timeline is difficult in our case without at the same time increasing the linguistic variation and adding many different genres. Thus, our focus is to run most of the experiments using texts that temporarily are located relatively close to the era of Seneca the Younger and the same kind (in verse).⁵ Second, there is the issue of the varying meter across the texts (e.g., iambic vs. hexametric), which constrains the vocabulary available to the author. For computational stylometry, different vocabulary means different features, and therefore dissimilarity between texts. While we cannot completely resolve this issue, we believe that we can limit its influence by considering patterns of very frequent character sequences rather than whole words (see subsection 4.1). In addition to that, prior work on cross-genre and cross-topic stylometry has empirically shown that character-based authorship attribution is robust to such variation (e.g., Stamatatos 2013, 343). On the one hand, it may be that this robustness also applies to the genre and meter variation in our case. On the other hand, it must be noted that since the disputed plays are compared to Senecan texts in the same genre and meter, while the imposter texts are in a different genre and meter, the likelihood of attributing the disputed plays to Seneca may be increased.

Table 10 in Appendix A provides a complete list of authors and texts included in the dataset variations used for each experiment. All works were obtained from the Perseus Digital Library (Crane 2024) except for Manilius’ Astronomica due to unavailability in the source.⁶ Thus, Astronomica was sourced from The Latin Library (Carey 2024).⁷

4. Feature Selection and Methods

The dataset was preprocessed and analyzed using the R package stylo (Eder et al. 2016) and The Classical Language Toolkit (CLTK)(Johnson et al. 2021).

4.2 Methods

All of the methods we employ estimate the stylistic similarity of texts as the distance between their features (i.e., character n-gram frequencies). For this, we pick the Cosine Delta distance metric because of its effectiveness in various test conditions and its particular effectiveness for inflected languages (Jannidis et al. 2015, 6–8; Evert et al. 2017, ii9–ii10; Eder 2022). Both the validation and main analysis phases utilize the 2,000 Most Frequent Character 4-grams (MFCs), a selection supported by studies indicating the performance of the Cosine Delta plateaus at this threshold for texts in Latin (Jannidis et al. 2015, 6–8; Evert et al. 2017, ii9–ii10).

In general, more MFCs lead to better performance since the features capture more stylistic variation. However, beyond the 2,000 MFCs, the character n-grams become rarer and are therefore not as informative. Therefore, we consider this point as adequate to capture the necessary amount of authorial fingerprint (Jannidis et al. 2015; Evert et al. 2017; Eder 2022). The frequency distribution plot (see Figure 3) illustrates this diminishing informativeness beyond the 2,000^th character 4-gram.

Figure 3: Frequency distribution of the character 4-grams in the whole corpus (i.e., 90 texts including the disputed plays). The vertical line is set to 2,000 to show that characters 4-grams after this threshold start to become quite infrequent. The result is what we expect to see since the distribution of the frequency of features in a given text follows Zipf’s law (the frequency f of a feature is inversely proportional to its rank r).

The study employs two exploratory analysis methods and one authorship verification method, presented in ascending order of robustness. Firstly, Principal Component Analysis (PCA) is applied. Secondly, the Bootstrap Consensus Tree (BCT) is introduced, followed by the General Imposters (GI) method, each briefly outlined in the subsequent sections.

4.2.1 Principal Component Analysis

PCA, a widely used unsupervised algorithm in authorship attribution and verification studies, reduces dimensionality by identifying principal components (eigenvectors) that explain feature variation. In this context, dimensionality refers to the number of features or variables initially present in the dataset (in our case the features that are generated by character n-grams). PCA helps reduce this dimensionality by transforming the data into a new set of variables, where each successive variable captures less and less of the total variance in the data. To preserve maximal data variance, PCA zeroes out smaller principal components, employing only those capturing the highest variance (VanderPlas 2017, 436). These components position texts in a two-dimensional visualization, enhancing readability for human interpretation but at the same time losing some of the variation information (Stamatatos 2009, 545). Similarity in frequency distribution correlates with spatial proximity in the PCA plot, indicating text dissimilarity based on vector dissimilarity. Closeness may reflect temporal proximity, common genre, or shared authorship (Manousakis 2020, 171–172). Isolated data points suggest the opposite. Applied exclusively to the Senecan corpus, PCA results use a correlation matrix due to its invariance to linear changes in units of measurement, making it suitable for scaled variables like relative frequencies of character 4-grams (Jolliffe and Cadima 2016, 6). The correlation matrix accommodates the varied scale changes within the broad range of 100 to 2,000 Most Frequent Character 4-grams (MFCs).

4.2.2 Bootstrap Consensus Tree

While the Bootstrap Consensus Tree (BCT) originates from the field of phylogenetics, it was introduced as a method for computational stylometry by Eder (2012) and has since been increasingly used to identify authorial and translator fingerprints (Rybicki 2012; Rybicki and Heydel 2013). The fundamental idea behind bootstrapping is to randomly select a large number of samples with replacements. This process allows us to average the estimates of these samples, thereby enhancing the recurrence of patterns within a document (Jurafsky and Martin 2024, 75–77). Moreover, this method assumes that frequent patterns will reappear many times (robustness), but by increasing the number of iterations and using the consensus strength, we incorporate a larger and thus more diverse number of patterns within a single text (diversity). In other words, a higher number of samples guarantees a greater variety of patterns, making the results more representative of the population.

To clarify some of the concepts mentioned in the previous paragraph: Sampling with replacement involves sampling units returning to the data pool, allowing them to appear in multiple data ‘snapshots.’ This facilitates the identification of frequently occurring patterns, but also risks letting outliers excessively impact results. To balance the influence of outlier impact, a large number of iterations is usually preferred (Kuhn and Johnshon 2016, 72–73). In addition, another concept that is being implemented in our approach to further balance the impact of outliers is consensus strength. Consensus strength means that patterns present only in a certain percentage of iterations will be included in the final result. For instance, if we have a consensus strength of 0.5 (i.e., 50%), then only patterns that appeared in at least 50% of the iterations will be included. Unlike a simple dendrogram, a key advantage of BCT lies in its consensus strength, ensuring that more reliable relationships above a specified threshold will influence the final output. Parameters utilized include an MFC n-grams range from 100 to 2,000 with a step of 100, and a consensus strength set to 0.5.

4.2.3 General Imposters Method

The GI method, initially introduced by Koppel and Winter (2014), won first place in the PAN competitions for shared tasks in authorship verification for two consecutive years (i.e., 2013 and 2014) (Seidman 2013; Khonji and Iraqi 2014). Since then, it has proven effective in authenticating disputed writings attributed to Julius Caesar, attributing the text Compendiosa expositio to Apuleius, and identifying the author behind the pseudonym Elena Ferrante (Kestemont et al. 2016; Stover and Kestemont 2016; Savoy 2020; Tuzzi et al. 2024).

In the context of the GI method, authentication involves determining whether a text is consistently attributed to an author across many comparisons and quantifying the confidence in this determination. Unlike many other authorship attribution methods, the GI method handles open-set authorship verification problems, allowing for scenarios where the actual author may or may not be among the candidates.

The GI method verifies authorship based on the document’s similarity to the purported author’s writings and dissimilarity with imposters. The process is akin to a witness identifying a suspect from a police lineup. Multiple iterations using different subsets of the 2,000 Most Frequent Character n-grams enhance the robustness of the results (Eder and Rybicki 2013). In each iteration, 50% of each imposter’s text and features are randomly selected for analysis, enabling consideration of numerous feature combinations and outlier detection, leading to more reliable outcomes (Eder et al. 2016). The method produces a score between 0 and 1 for each author in the lineup, indicating the proportion of times an author was identified. A higher score reflects greater confidence that the author wrote the disputed text (Eder 2018). This score not only gauges stylistic similarity, but also assesses how consistently an author is identified as opposed to imposters.

5. Validation

The methods described were assessed across multiple validation sub-corpora (detailed in respective subsections) to measure their efficacy for authorship attribution/verification tasks. Utilizing the Cosine Delta distance metric and a frequency band of the top 2,000 MFCs 4-grams, no culling parameter was applied to ensure an adequate feature set.⁹

5.1 PCA (Validation)

To validate PCA, a sub-corpus was created from the initial dataset, consisting of works by four authors: Ovid, Lucan, Persius, and Statius (see Table 10). These authors were chosen due to their temporal proximity to Seneca’s work, despite differences in genre; while Lucan, Ovid, and Statius wrote epic poems, Persius focused on satires. Including Persius’s works in this validation corpus was based on their relatively smaller size compared to the other works, posing a potential challenge for PCA analysis.

Demonstrating the method’s emphasis on text variance over author names, three texts had their author names replaced with “unknown.” The filenames were adjusted to unknown_amores for Ovid’s Amores, unknown_theb_1 for Statius’ first book of Thebaid, and unknown_sati_4 for Persius’ fourth Satura. The first two texts were randomly chosen, while the last, due to its small size (392 tokens, including pronouns), posed a challenge for PCA.

Figure 4 presents the PCA results using a correlation matrix, showcasing the impact of different frequency bands (100 MFC 4-grams in Figure 4 and 2,000 MFC 4-grams in Figure 4). Observation reveals a consistent attribution in both cases, with larger frequency bands showing less distinct clusters. Notably, in Figure 4, Persius’ fourth Satura and Ovid’s text Medicamina Faciei Femineae exhibit some movement outside their relevant clusters. This deviation could be attributed to the small size of these texts relative to others in the corpus, as text size may influence authorship attribution or verification tasks (Luyckx and Daelemans 2011, 52; Eder 2013, 180).

Figure 4: PCA using a correlation matrix to visualize the results. Figure 4a demonstrates how the attribution works given a small frequency band (i.e., 100 MFCs 4-grams). Figure 4b demonstrates the authorship attribution given a larger frequency band (i.e., 2,000 MFCs 4-grams).

Someone might ask at this point why the two PCAs, even though they use exactly the same corpus, differ in terms of variance they capture. The reason is that 100 MFC n-grams might capture more variance than when we include less frequent features. With 100 MFC n-grams, the dimensionality is lower, thus PCA more easily captures the variance explained by the features. With the addition of more features, more noise might be added and thus less variance might be captured (see Figure 4).

5.2 BCT (Validation)

At this point, it is crucial to note that the Bootstrap Consensus Tree (BCT) functions as a consensus, capturing more dimensions and information than PCA due to the robust patterns observed across different iterations (see subsubsection 4.2.2). Unlike PCA, which reduces the data to a few principal components by focusing on the largest sources of variance, BCT aggregates information from multiple bootstrap samples, thus integrating a wider range of variations and subtleties in the data. This allows BCT to provide a more comprehensive view of the data’s structure and relationships, encompassing nuances that PCA might overlook.

In this validation, the corpus is slightly changed and the file names are altered again to demonstrate the independence of the final result (unrooted tree and branches) from the file names. Due to its very small size, this time instead of Amores we use Medicamina Faciei Femineae as part of the unknown texts by converting its filename to unknown_medicam. The rest of the “unknown” texts remain consistent as in the previous validation test (see subsection 5.1).

All texts in the test set were accurately attributed to their respective authors using BCT (see Figure 5). Notably, the texts renamed as “unknown,” which presented challenges in PCA (i.e., Ovid’s Medicamina Faciei Femineae and Persius’ 4^th Satura) were handled adeptly by BCT, emphasizing the robustness of BCT in authorship attribution tasks regardless of text size (see subsubsection 4.2.2 for further details).

Figure 5: A BCT that was generated using the top 100-2,000-100 (start-end-step) MFC 4-grams and Cosine Delta as distance metric (no culling set); pronoun culling was applied and a consensus strength of 0.5 was used.

5.4 Validation Findings

The validation indicates effective performance for all methods on the texts within the corpus, with PCA showing limitations for short texts (Figure 4). The BCT method demonstrates robust recognition of authorial fingerprints across varied text lengths, owing to their bootstrapping techniques, culminating in a consensus from multiple iterations (see Figure 5). Similarly, the GI method commits only one mistake (see Figure 6): Martial is being attributed as the author of the 1^st Satura of Persius with a confidence score of 0.71 (+0.04 of p1). These findings suggest that the selected frequency band (top 100 to 2,000 Most Frequent Character 4-grams) is informative for capturing authorial fingerprints, yielding high success rates in each validation scenario. Consequently, the main analysis phase will replicate this process with a focus on the disputed texts.

Figure 6: The confusion matrix shows the results of the GI method on the validation dataset. P1 value = 0.31 and p2 value = 0.67. The result is based on the author that returned the highest score for a given text. The two disputed plays, Oct. and H.O., by Seneca the Younger are excluded from the validation set.

6. Results and Discussion

We first explore the stylometric properties of the Senecan plays using PCA to see how they relate to each other. When treating the plays as a whole, it can be observed that from the two disputed texts, only Oct. behaves as outlier within the Senecan corpus of plays (see Figure 7). However, H.O. consists of 11,1147 tokens, which is almost twice as many tokens as the average size of a Senecan play (excluding Oct.) (x̅ = 6192.5 tokens). When H.O. is divided into two halves to align its size more closely with the average size of a Senecan play, it shifts away from the cluster of Senecan texts (see Figure 8). Meanwhile, Oct. remains consistently outside the cluster of Senecan plays. A possible explanation of why Oct. and H.O. behave as outliers is the fact that when considering the works of a single author using a PCA, the genre-related signal tends to become stronger than the author-related signal (Stover and Kestemont 2016, 659).

Figure 7: PCA correlation matrix of the Senecan corpus of plays (disputed and not). The texts seneca_oct and seneca_her_o correspond to Oct. and H.O., respectively. In both cases, regardless of the size of the frequency band, Oct. and Phoenissae behave as outliers within the Senecan corpus, whereas H.O. is placed among the Senecan plays. It is important to highlight that the percentage shown in PC1 and PC2 varies in each plot because the principal components capture different amounts of variance each time.

Figure 8: PCA correlation matrix of the Senecan corpus of plays (disputed and not), this time with H.O. split in half. H.O. starts to behave as an outlier and Oct. remains among the outliers. It is important to highlight that the percentage shown in PC1 and PC2 varies in each plot because the principal components capture different amounts of variance each time.

In addition to that, it should be stressed that in all of the PCA plots Phoenissae also behaves as an outlier within the Senecan corpus, while its authorship is not disputed. An explanation for this behavior could be that Phoenissae is an unfinished play and the shortest text in the Senecan corpus of plays. Furthermore, the aforementioned play has a lot of issues in terms of structure and unity. Based on the number of innovations that were attempted in the text, Frank (2018, 1–2) points out that this might be the reason why this text was abandoned by Seneca when he realized the difficulty of this venture.

Figure 9 shows a Bootstrap Consensus Tree (BCT) for the Senecan plays alongside two selected authors from the literature of the Early Empire, namely Lucan and Statius. Statius is included to test the hypothesis of Ferri (2003, 17–27), which suggests a temporal connection between the composition of Oct. and Statius. On the one hand, the BCT exhibits distinct branches for each author, placing both disputed plays in proximity to the Senecan works, but Oct. gravitates slightly towards the center of the unrooted tree. This again highlights the special nature of this specific text. On the other hand, H.O. remains in the Senecan cluster of plays.

Figure 9: BCT of texts from Statius (Achilleid, Thebaid, Silvae), Lucan (Pharsalia), and Seneca (plays). The texts seneca_oct and seneca_her_o correspond to Oct. and H.O., respectively.

Regarding the GI method, we test five different scenarios. Since GI returns a confidence score as the final output, we need to pick thresholds for when to reject or accept the verification of an author. Stylo provides a method to automatically determine such thresholds using cross-validation (the stylo.optimize() method). For Scenario 1 it gives p1 = 0.45 and p2 = 0.55, for Scenario 2 the threshold is set to the values p1 = 0.22 and p2 = 0.76, whereas for Scenario 3 it returns p1 = 0.00 and p2 = 0.98 (for a brief description of the different GI scenarios see Table 3); below p1 Seneca is not the author; above p2 Seneca is identified as the author; when the score is in between, no determination can be made. Unfortunately, the cross-validation method is too expensive to run on the larger datasets we use in the rest of our experiments (see scenarios 4 and 5 in Table 3) due to the nested loops and the bootstrapping that takes place, which increases the time complexity of the algorithm. Therefore, we will use a conservative threshold of 0.9 for all our experiments.

Since there is no option to set a random seed in stylo’s environment, we applied a workaround to achieve more consistent and replicable results. Specifically, we employed the GI method by running the analysis 10 times, with each run consisting of 100 iterations. We report both the average score and the standard deviation across the 10 runs. This approach allowed us to assess the stability of our results by observing variability and central tendencies over multiple independent runs, even without a fixed random seed. A drawback, however, is the increase in time complexity, as this method adds an external loop of 10 runs to the existing internal loop of 100 iterations, resulting in a total of 1,000 iterations.

With the GI method, Scenario 1 and 2 confidently attribute Seneca the Younger as the author of the disputed plays (see Table 3). Next, in Scenario 3, we consider the cento-argument by Ferri (2014, 48).¹⁰ We do this by identifying and removing sentences from the disputed texts resembling those in the Senecan corpus of plays. We operationalize sentence similarity using tf-idf (term frequency-inverse document frequency) vectors of the character 4-grams for each sentence, and cosine similarity as the metric for the similarity of pairs of sentences. We identify and exclude all sentences with a similarity exceeding a threshold of 0.6. The cosine similarity metric measures directional similarity between vectors, irrespective of magnitude or scale (Singhal 2001, 2–3). The presented methodology, when integrated with specific preprocessing procedures including the conversion to lowercase, elimination of punctuation marks (with the understanding that an editor may subsequently reintroduce punctuation marks), and the utilization of character 4-grams as distinctive features, exhibits the capability to discern similarities. This capability is exemplified in Table 4, wherein similarities are identified not only among various declensions of identical terms but also amid permutations in word order. For Oct. from a total of 422 sentences, we identified and thus removed 2 (i.e., 0.46%) sentences above the similarity threshold (i.e., 0.6), whereas for H.O., from a total of 1149 sentences we identified and removed 33 (i.e., 2.87%) sentences.

To address potential length bias and investigate possible mixed authorship throughout the disputed texts, in Scenario 4 each text exceeding 500 tokens is divided into non-overlapping chunks of 500 tokens. This approach, inspired by Rolling Stylometry (Eder 2016), simplifies the process by using non-overlapping segments instead of overlapping ones. Rolling Stylometry works by analyzing text in sequential segments to track stylistic patterns and changes throughout a document or corpus. The results for Scenario 4 (Figure 10 and Figure 12) reveal a nuanced internal composition, uncovering authorship diversity within the disputed plays. Although Seneca’s authorship dominates, specific segments warrant attention, as highlighted in Figure 10 and Figure 12.

Figure 10: Results of the GI method for Oct.’s chunks using the corpus of texts in verse (Scenario 4). The dots represent the average score across ten runs, while the error bars indicate the standard deviation, reflecting the range of score variability observed throughout the entire process.

When we compare the chunks of Oct. with the chunks from the main corpus in Scenario 4, we see that chunks 1, 2, 3, 6, 8 are below the threshold of 0.90 (see Figure 10 and Table 5). However, for chunks 1, 2, 6, and 8, we obtain an average confidence score of 0.85, 0.83, 0.88, and 0.87, respectively, which indicates that Seneca authored these parts, but someone might have made minor changes. Focusing on chunk 3 (0.31), there are features that appear merely as a decoration (e.g., the change between iambics and anapaests) and parts of the text that are a critique of Nero (e.g., the rhetorical question of why Jupiter is killing the innocent and not those who deserve it, i.e., Nero; see Ferri (2014, 180–210)). Criticizing the emperor himself, especially by someone (Seneca) who was his advisor, was not an easy task.

In Scenario 5, the comparison reveals one more chunk of Oct. that might raise concerns (see Figure 11 and Table 7), chunk 6 (0.57). The playwriter here condenses the time in a way that seems unnatural for Seneca the Younger, in order to present a large number of events in a small amount of time (Ferri 2014, 307–309). The rest of the chunks (i.e., chunks 1, 2, 11) do not fall far from the threshold of 0.9, indicating that someone might have made minor adjustments.

Figure 11: Results of the GI method for Oct.’s chunks using the dataset of Kestemont et al. (2016) (Scenario 5). The dots represent the average score across ten runs, while the error bars indicate the standard deviation, reflecting the range of score variability observed throughout the entire process.

Regarding H.O. both scenarios 4 (see Figure 12 and Table 6) and 5 (see Figure 13 and Table 8) reveal a pattern where the first half of the text is securely Senecan, and the second half shows chunks that do not meet the threshold. These results align to some extent with the hypothesis that the first half of the text originates from Seneca, while the remainder may have been finished by someone else (Tarrant 2017, 97). However, most of the chunks in the second half still have high scores, suggesting that the second half is a case of mixed authorship, rather than being written completely by someone else.

Figure 12: Results of the GI method for H.O.’s chunks using the corpus of texts in verse (Scenario 4). The dots represent the average score across ten runs, while the error bars indicate the standard deviation, reflecting the range of score variability observed throughout the entire process.

Figure 13: Results of the GI method for H.O.’s chunks using the dataset of Kestemont et al. (2016) (Scenario 5). The dots represent the average score across ten runs, while the error bars indicate the standard deviation, reflecting the range of score variability observed throughout the entire process.

Lastly, note that for both Scenarios 4 and 5, the standard deviation of the text chunks that fall under the threshold of 0.9 tends to be larger than the standard deviation of the chunks above the threshold. This is another indicator that these chunks differ stylistically and warrant further study.

7. Conclusions

Our findings underscore the complexity of the authorship verification problem, particularly evident in the case of the disputed Senecan plays, Oct. and H.O.. Across experimental runs, varying results highlight the intricate nature of this challenge in computational stylometry.

Paraphrasing Stover and Kestemont (2016, 647), our aim is not to replace existing modes of analysis, but rather to illuminate longstanding issues by shedding new light through the application of innovative tools grounded in traditional methods. This analysis underscores the importance of considering genre and meter variations in our conclusions. As previously noted, these two factors can introduce complexities due to their influence on vocabulary. It is impossible to completely remove the influence of variation in meter and genre (Corbara et al. 2023), thus we employ preprocessing techniques to mitigate their impact on the final results.

Through the validation phase, we demonstrate the effectiveness of these techniques for our task. Consequently, we apply these techniques consistently to generate uniform features for each method. Notably, in the case of the two exploratory methods – PCA and BCT – Oct. and H.O. emerge as intriguing examples of texts concerning their authorship among the Senecan corpus of plays. In certain instances, they exhibit clustering with the broader set of Senecan plays, while in other cases they do not. For instance, when using only the Senecan plays, the genre and thus the meter and the size of the plays seem to win over the authorial fingerprint (see the cases of Phoenissae and H.O. in Figure 7).

The initial two scenarios of the GI method confidently verify Seneca as the author with a high degree of confidence (= 1.0). Moreover, after removing sentences from both disputed plays that are similar to sentences from other Senecan plays, the GI method still verifies Seneca as the author of the disputed plays. Therefore, the stylistic similarity of the disputed plays to the works of Seneca cannot be explained by borrowed phrases. Nevertheless, the fourth and the fifth scenarios highlight segments in Oct. and H.O. that are unlikely to be attributed to Seneca, implying the involvement of a distinct author or editor. Concentrating on H.O. (see Figure 12 and Figure 13), we posit that an editor of the text may have edited or added certain parts in the original play, even though it was primarily authored by Seneca. Lastly, the results hold up when the disputed plays are compared with a larger corpus of prose texts, suggesting that our findings are robust.

Against this algorithmic confidence, two objections can be made: First, we cannot rule out a highly skilled imitator; however, this seems implausible given the advanced nature of modern stylometry, of which an imitator could not have been aware. Second, the distractor texts differ in genre and meter from the Senecan texts. Unfortunately, due to the limitations of extant texts, it is impossible to construct a perfect distractor corpus. Therefore, while our empirical findings cannot positively confirm Seneca as the author of the disputed plays, our main contribution is that, perhaps contrary to expectations given the consensus against Seneca’s authorship, most of the text of the disputed plays is highly stylistically similar to Seneca’s writings. This means that Seneca cannot be ruled out as the author of the disputed plays based on stylometry. Moreover, our results provide evidence for mixed authorship in specific parts of the disputed plays.

8. Further Research

Deciphering the authorial fingerprint of the Senecan disputed plays requires further investigation and consideration of study limitations. Future work could take a closer look at the specific text chunks diverging from Seneca the Younger’s style. Employing Rolling Stylometry or using the General Imposters method with overlapping text segments (Eder 2016; Beullens et al. 2024), together with close reading approaches, could enable identification of authorship at the sentence level and enhance understanding of why these segments differ from Seneca’s style. Moreover, exploring the impact of prosody in ancient languages (e.g., Latin or Ancient Greek) on stylometric methods is another avenue for investigation. Controlled experiments using authors who wrote in different meters would make it possible to quantify their effect on the stylometric profile of texts. Furthermore, while the GI method has been shown to be robust and reliable in previous studies, including for Latin (Kestemont et al. 2016), it would be useful to examine and empirically test whether an imitator can successfully deceive the GI method. The Ferrante case shows that the pseudonym of an author who is highly motivated to hide his or her identity can be unmasked by pinpointing the gender, age, region, and city of the author profile (Mikros 2018). A potential improvement would be to use a large language model that could also detect paraphrases by taking into account semantic similarity.

9. Data Availability

Data used for the research can be found at: https://github.com/PaschalisAg/seneca_stylometry. It has been archived and is persistently available at: https://doi.org/10.5281/zenodo.14002368.

10. Software Availability

All code created and used in this research has been published at: https://github.com/PaschalisAg/seneca_stylometry. It has been archived and is persistently available at: https://doi.org/10.5281/zenodo.14002368.

11. Acknowledgements

We extend our sincere gratitude to all the reviewers for their invaluable feedback. Their insightful comments illuminated aspects of the paper that might otherwise have escaped our notice. We are grateful to Bram van der Velden for feedback on a draft. Special thanks are due to Vasileios Dimoglidis, a PhD candidate in Classics at the University of Cincinnati (UC), whose initial inspiration sparked this study. Additionally, we are grateful to Associate Professor Vasileios Pappas and Professor Helen Gasti from the University of Ioannina (UOI) for generously providing an extensive bibliography to support our research into the non-quantitative approaches examined in this study.

12. Author Contributions

Paschalis Agapitos: Code, Data curation, Conceptualization, Investigation, Writing – original draft

Andreas van Cranenburgh: Conceptualization, Formal analysis, Methodology, Supervision, Writing – review & editing