Comparing groups of texts to identify what is distinctive about each is a fundamental approach in many research contexts. In computational literary studies, such comparisons are particularly valuable for exploring literary style, genre conventions, authorial voice, or historical shifts in discourse. Such contrastive analyses of literary corpora have, for instance, been used to study different characters’ speech in Shakespeare’s plays (Culpeper 2009), in the context of gender markers in English-language novels (Weidman and O’Sullivan 2018), for determining the place of tragicomedy with respect to comedy and tragedy (Schöch 2018), or for identifying phrasal expressions characteristic of subgenres of the contemporary French novel (Gonon et al. 2018).

A key challenge in this task, alongside selecting the appropriate comparison corpora, is finding the most suitable measure and parameters for a specific research question and corpus composition. There is a wide range of measures available, and the list of most distinctive features they identify can vary considerably (as shown e.g. by Du et al. 2021a for the Zeta and Eta measures). While in principle, virtually any countable feature of texts may be submitted to a contrastive statistical analysis in order to identify distinctive features, we focus exclusively on lexical features in this research, specifically on word unigrams. In this paper, we explore and evaluate various measures of distinctiveness, also known as keyness measures, which support such research from a quantitative perspective. Although we do not prescribe a particular measure for researchers to use, our paper offers valuable insights into the characteristics of these measures, helping researchers understand their behavior and the potential outcomes when applying different distinctiveness measures in their studies.

In the research we report on here, we focus on evaluating measures of distinctiveness through an analysis based on synthetic texts.1 Our research proposes a new method for evaluating measures of distinctiveness, utilizing synthetically created text collections that reflect word frequencies as they would have occurred in a regular corpus built from the same original texts. Studies based on naturally occurring language must work around the fact that the frequency and dispersion of any word will vary and correlate to some extent. Our approach allows for precise, independent manipulation of word frequency and dispersion by inserting an artificial word. By conducting keyness analysis using synthetically created datasets and through inserting an artificial word with precisely manipulated frequency and dispersion into the synthetic dataset, we aim to systematically uncover the characteristics of different measures. Our goal is to determine the degree of sensitivity of each measure to variations in frequency and dispersion. Our method enables us to uncover new advantages and limitations of distinctiveness measures and to compare their sensitivity to frequency and dispersion variations under consistent conditions.

The structure of our paper is as follows: We begin with an overview of previous work in the evaluation of measures of distinctiveness (section 2). Next, we describe our dataset (section 3) and provide a detailed explanation of our methodology (section 4). We then outline our hypotheses (section 5) and present the results of our evaluation (section 6). Finally, we conclude by summarizing our key findings and discussing potential directions for future research (section 7).

Evaluating measures of distinctiveness is challenging due to the fact that generating a gold standard annotation, based on which performance measures such as precision and recall can be calculated, is very difficult. Distinctiveness is not an inherent characteristic of a word, nor does it depend only on local context; rather, it can only be detected in the context of the entire target corpus while considering it in comparison to another corpus. Therefore, alternative methods of comparison and evaluation of the measures of distinctiveness are required. To tackle this challenge, several studies have attempted to evaluate distinctiveness measures using various methods.

Kilgarriff (2001) examined corpus similarity by reviewing the mathematical characteristics of various distinctiveness measures and argued that the Chi-squared test is the most suitable in finding the most characteristic words of a corpus. Paquot and Bestgen (2009) compared three different measures in their ability to identify frequent and well distributed keywords of academic prose as opposed to fictional prose and discovered that the t-test leads to the best results for their task. Lijffijt et al. (2014) explored a broad array of measures, focusing on the statistical characteristics of these measures to identify their sensitivity to differences in word frequencies and distributions. The authors randomly sampled a text corpus into two parts in order to minimize differences in both parts and then performed a test for uniformity of p-values. Egbert and Biber (2019) introduced a distinctiveness measure based on dispersion, combining a straightforward dispersion metric with a log-likelihood ratio test. They compare the effectiveness of this approach with corpus frequency methods for identifying distinctive words in online travel blogs. Their study demonstrates that the dispersion-based measure outperforms the other types of measures. Sönning (2023) evaluated 32 metrics, categorized into four dimensions of keyness. Like previously mentioned researchers, he distinguished between two primary perspectives on keyness: frequency-based and dispersion-based measures. His study assessed the effectiveness of these metrics in identifying predefined key verbs in academic writing. The results reveal significant differences among the metrics, with the Wilcoxon rank-sum test and dispersion-based measures emerging as the most effective.

The research we report on here also builds on fundamental work on measures of distinctiveness by our Zeta and Company project group. We conducted an in-depth analysis of the qualitative characteristics of these measures (Schröter et al. 2021). To enhance accessibility and usability, we implemented nine measures of distinctiveness in the Python package pydistinto (Du et al. 2021b). With Du et al. (2021a), we then introduced a new dispersion-based measure called Eta and compared it with the existing Zeta measure to highlight the advantages and disadvantages of each. Our group also performed a quantitative evaluation of nine measures on natural texts, including several dispersion-based measures, using a downstream classification task (Du et al. 2022). Our approach involved first identifying a given number of distinctive words provided by each measure for novels of a specific genre, in comparison to other literary genres. These distinctive words were then used to classify the novels by genre, with the classification accuracy obtained being a measure of each word list’s distinctiveness (in the qualitative sense of discriminatory power). We concluded that dispersion-based measures are more effective than frequency-based measures in identifying characteristic words of a target corpus.

Overall, while previous studies have provided valuable insights into distinctiveness measures, their reliance on abstract statistical analyses, intuitive evaluations, or a narrow selection of measures underscores the need for further research. Our study addresses these limitations by introducing a controlled, synthetic approach with precise manipulation of word frequency and dispersion, while also incorporating a wide range of different measures to enable a more systematic and nuanced assessment of their sensitivity. We have already conducted several analyses using naturally-occurring texts. Now, with our approach using synthetic texts, we aim to test theoretical insights about the measures under specially controlled conditions, allowing for a clearer understanding of how each distinctiveness score is calculated.

We think that using a wide variety of evaluation strategies is most likely to result in robust results, as past experience has shown that even theoretically sound and convincing arguments may not hold up to empirical scrutiny, whether quantitative or qualitative (as a case in point, consider investigations of distance-based stylometric authorship attribution; Argamon 2007Evert et al. 2017).

Our research is conducted on a synthetic text collection generated through random sampling at the word level from a corpus of French contemporary novels. The foundation for this corpus is a balanced subset from our larger collection of French contemporary popular novels and consists of 320 novels from the 1980s and 1990s. This custom-built corpus maintains equal representation (in terms of the number of novels included), per decade and across four subgroups: literary fiction, sentimental novels, crime fiction novels, and science fiction novels.

The original text corpus comprises approximately 19 million words. We load the entire corpus as a single dataset and randomly sample synthetic ‘novels’, each with a consistent length of 40,000 words. The sampling was performed at the word level. Our newly generated corpus contains 320 synthetic ‘novels’, matching the number of novels in the original corpus. This approach addresses two main objectives. First, it ensures that the generated corpus reflects the word occurrences and frequencies as they can be observed in the original corpus. Second, it results in a homogeneous corpus, purposefully eliminating subgenre differences because each text is sampled from the entire corpus.

The objective of our analysis is to assess the hidden properties and limitations of the measures of distinctiveness in identifying distinctive words. This is achieved by applying each measure to a homogeneous synthetic corpus to which an artificial word with a controlled frequency and dispersion has been added. Systematically varying the frequency and dispersion of this word, and observing how its keyness rank in the results varies as a result, shows us to what degree a given keyness measure is sensitive to differences in frequency and/or dispersion. We chose to compare the ranks, rather than the measures’ scores, for better comparability.

In our analysis, we have analyzed all nine measures of distinctiveness implemented in our Python package pydistinto. The following measures are available in this package: Burrows Zeta (Burrows 2007), logarithmic Zeta (Schöch et al. 2018), Eta (Du et al. 2021a), TF-IDF (Spärck Jones 1972), Wilcoxon rank-sum test (Wilcoxon 1945), Welch’s t-test (Welch 1947, Mann and Whitney 1947), the Ratio of relative frequencies (RRF, Gries 2010), the Chi-squared test (Plackett 1983), and the Log-likelihood ratio test (LLR, Dunning 1993).2 The implemented measures can be categorized into three distinct groups based on their approach to identifying unique keywords when comparing a target and a comparison corpus. Within this framework, the techniques employed can be classified as follows:

Our approach was as follows: As pydistinto requires a certain format of input data (CSV format including the following columns: token, lemma and POS), the original French corpus was annotated with spaCy before randomization.4 For the analysis with pydistinto, we used lemmas as the feature type. At the beginning of the process, the synthetic corpus was divided into segments of equal length, each containing 5,000 words, resulting in 8 segments per novel and a total of 2,560 segments. This segmentation is essential for the calculation of certain measures, such as Zeta and Eta.

Subsequently, the entire corpus was randomly divided into two sub-corpora of equal size for each run of pydistinto: target and comparison corpus. An artificial word was then added to both the target and comparison corpus parts with a specified frequency and dispersion.5 To maintain a constant total word count while adding an artificial word, each instance of the artificial word replaces one instance of an existing word in the corpus.

Our experiment was conducted in two primary settings to investigate the impact of two criteria – the frequency and dispersion of the artificial word within a corpus – on its distinctiveness score, calculated by different measures.

In the first setting, we added an artificial word to only one segment of the target and comparison corpus, albeit with varying frequency. This setting enables us to analyze the influence of only one parameter, namely the frequency. The frequency of the artificial word was set to 10 in the comparison corpus and remained constant there, while varying from 10 to 2,000 words in the target corpus. We used 12 different parameters for the frequency setting in the target corpus (10, 20, 30, 40, 50, 100, 200, 300, 400, 500, 1000, and 2000). For each parameter setting, pydistinto was run 100 times to mitigate the impact on the results of high scores for frequent words, which may arise as a result of variation that follows from the random sampling procedure and may in turn influence the distinctiveness score of an artificial word. The corpus was randomly divided into target and comparison parts at the level of the ‘novels’ for each run. Given the fact that texts were built by randomly sampling words from the entire corpus, and the two subcorpora were built by randomly sampling ‘novels’ from among all ‘novels’, any difference between the target and comparison corpora, apart from the artificial word, can only be due to random variation.

In the second setting, we experimented with the dispersion of the artificial word. In this case, the frequency of the artificial word was kept constant at 1,000 occurrences in both the target and comparison corpus, but its dispersion varied in the target corpus while remaining constant in the comparison corpus. The idea was again to isolate one parameter, in this case dispersion, and analyze its influence on the performance of the different measures. For the comparison corpus we used the following settings: we added 1,000 instances of the artificial word to just 1 segment.6 Dispersion variation was achieved by adding the artificial word with a specified, constant total frequency to the target corpus, but with varying degrees of dispersion. We conducted distinctiveness analyses with variations in the target corpus according to the following schema, where the first number refers to the number of segments that receive the artificial word, and the second to the number of times the artificial word is included in each of the selected segments: 1/1000, 2/500, 5/200, 10/100, 20/50, 50/20, 100/10, 200/5, 500/2, 1000/1. The product of the two values, and therefore the total frequency, remains constant at 1,000 (and is therefore identical to the frequency of the word in the comparison corpus), but the number of segments these occurrences are spread out over is varied systematically. This resulted in a total of 10 parameter settings for the dispersion experiments. Again, pydistinto was run 100 times for each parameter setting.

Following this step, the results for each parameter setting were combined into a single dataframe. Subsequently, all words in the corpus were sorted based on their distinctiveness scores, and for each measure, the rank of the artificial word following from its distinctiveness score was recorded. Each measure’s performance was evaluated based on the rank of the artificial word (where a rank of 1 indicates the highest distinctiveness score).

For this evaluation experiment, we developed the following hypotheses:

Because our corpus is based on naturally occurring word frequencies, we conducted an additional analysis to identify potential artifacts caused by random sampling effects in the synthetic texts without the artificial word. This analysis aimed to identify the frequency differences of words in the corpus across multiple runs.

Figure 1 illustrates the relationship between rank and the Ratio of Relative Frequencies (RRF) scores, based on 100 runs of randomly sampled synthetic corpora. As shown, the first rank is typically achieved with RRF scores ranging from 10 to 18. This suggests that, due to the natural variations in the frequencies of existing words, an RRF score below 10 for the artificial word is unlikely to secure the first rank.

As discussed in section 4, we conducted our evaluation in two main settings: frequency variation of an artificial word and dispersion variation. First, we are going to discuss the results of the evaluation based on frequency variations, before moving on to the results for dispersion variations.

Conducting analyses of measures of distinctiveness based on synthetic texts, we created ideal conditions to uncover the hidden properties of a range of such measures. Through our experiments, we tested the sensitivity of these measures to variations in the frequency and dispersion of a specific word. In many cases, our hypotheses regarding the performance of the measures were confirmed. Frequency-based measures are not sensitive to variations in dispersion, while dispersion-based measures are not affected by frequency variations. These observations are not surprising, of course, but they do validate our method.

However, some hypotheses were partly rejected and we have also uncovered some previously unknown (or at least undocumented) properties of measures of distinctiveness. In particular, we found that LLR and chi-squared tests are even more sensitive to frequency variation than RRF. For this reason, we generally do not recommend using the LLR and chi-squared tests, as they are highly sensitive to changes in frequency and are therefore not well-suited for keyness analysis aimed at identifying important content words. Both Zeta variations and the rank-sum test demonstrated similar scores and abilities to detect distinctive words, including cases in which the differences only concern the dispersion of words. Moreover, we discovered that TF-IDF is highly sensitive to dispersion differences of the target word, compared to other dispersion-based measures. Finally, we found that Eta cannot detect a word with a clear contrast in dispersion when its frequency is the same in both the target and comparison corpora. In our evaluation we observed words steadily moving up in rank with Zeta and rank-sum, while TF-IDF and Eta show more abrupt increases. We suggest that a gradual, continuous rank improvement is a desirable characteristic of a distinctiveness measure, as it indicates better sensitivity to slight variations in dispersion and is likely to produce more predictable results. For example, if a researcher is interested in identifying words that display contrasting dispersion within two subcorpora, without considering their frequency, then Zeta and the rank-sum test would be most appropriate for this task.

Despite the interesting observations derived from these analyses, there is significant potential for future work. One key step is to extend our framework by implementing additional measures of distinctiveness. Another area for future work involves expanding our analysis by implementing additional parameter settings that combine frequency and dispersion variations of the artificial word. Isolating dispersion or frequency often results in constant scores from the measures, but combining these parameters promises to provide new opportunities to uncover additional properties of these measures. A final, crucial step is to explore practical applications of this newfound knowledge about distinctiveness measures. Understanding the specific contexts and scenarios in which each of these measures can be most effectively utilized will open up new possibilities and enhance our ability to analyze and compare textual corpora more accurately.