The most important direction of innovation in natural language processing (NLP) of the last ten years has been the development of distributional semantics into a framework which underlies almost all processing models in NLP. This idea reached its current form through four major phases.

In the first phase, distributional semantics, built on ideas from structuralist semantics (Firth 1957), was developed into an approach that aimed at capturing the meaning of target words by counting co-occurrences of these target words with words in their context. This naturally corresponds to a representation of words as context vectors (or embeddings), and of semantic similarity as a measure of contextual similarity between words (Miller and Charles 1991).

In the second phase, the construction of such embeddings moved from explicit context counting to the use of neural language models. Such language models are presented with a textual prefix (‘holiday on Sri …’) and have the task of predicting the subsequent token (‘Lanka’). When trained to do so on large text corpora, the learned parameters of such language models are essentially word embeddings (Mikolov et al. 2013).

The next phase involved the introduction of the so-called transformer architecture, which introduces the ability to make word embeddings sensitive to their context: e.g., ‘root’ receives a different embedding in the phrase ‘root canal’ than in the phrase ‘square root’. The resulting contextualized language models can account for lexical ambiguity in a principled fashion (Devlin et al. 2019). Model families of this type, such as BERT, have shown to be applicable to almost all standard tasks in NLP by obtaining word embeddings for an input text, aggregating these embeddings to obtain a text embedding (e.g. simply by averaging), and passing the aggregate through a very simple decision mechanism – e.g., a single-layer neural network with two output units which directly corresponds to a logistic regression model over the text embedding. Such models are typically trained in two phases: First the language model is ‘pre-trained’ on a large amount of unlabeled text, then the decision layer is ‘fine-tuned’ on a comparatively small amount of task-specific labeled text.

The final phase in the development of language models consists in the emergence of large language models (LLMs) which result from massive scaling up of the model training. The resulting LLMs are to a substantial degree task-agnostic (i.e., do not require fine-tuning) and can be instructed (‘prompted’) directly to carry out some task (P. Liu et al. 2023). To solve a classification task, for example, models can be asked to answer ‘yes’ or ‘no’ similar to the way a human annotator would be instructed.

For the task we consider in this study – predicting the gender of a character in a drama based on (some part of) their utterances – both contextualized embedding models and LLMs are plausible candidates to develop classifiers. We decided to build our classifier on top of a contextualized embedding model. Such models are typically much smaller (and thus computation efficient), they are more transparent (the details of the training processes of almost all LLMs are confidential (Liesenfeld et al. 2023)), and prompt-based approaches face problems regarding robustness and bias (Webson and Pavlick 2022).

As outlined above, a central question that must be decided when presenting text to a LM for classification is the length of the input for which the model is supposed to make a single prediction, that is, the input granularity. Short inputs may consist of utterances which are not informative regarding the speaker’s gender. As for long inputs, older LM-based models have generally a limited input size (often 512 tokens)1 and while newer LLM-based models can deal with much longer inputs, they struggle with identifying the relevant information, in particular if it is in the middle of the text (N. F. Liu et al. 2024).

Figure 1 shows a simplified view of the hierarchical structure of a character’s utterances in a drama: At the bottom we have individual utterances, which vary in length from single sentences to long monologues. Utterances can be grouped together at the scene level, and then again at the global character level. Models can be trained to make gender predictions at each of these levels (shown in blue and red, respectively), and predictions can vary in confidence (shown by saturation). Since we do not know a priori which of these levels works best, our experiment will evaluate models that have been trained at each of these three levels.

If we choose to classify at a more fine-grained level (such as utterances), there is no guarantee that all utterances of one character are classified in the same way – in fact, given our assumption that at least some utterances are uninformative regarding gender, complete consistency would be surprising (utterances 2 vs. 3 in Figure 1). Therefore, we also need to define an aggregation method to return a character-level response based on the concrete utterance- or scene-level predictions.2 We consider two aggregation strategies. The simple one is majority vote: We count the number of utterances (or scenes) predicted as male or female, respectively, and assign the more frequent gender. In Figure 1, 3 out of 4 utterances are classified as male, which leads to an overall classification of the character as male. The second strategy makes use of the fact that LM-based models do not just output labels, but also probabilities. Assuming that more informative inputs are classified more confidently, taking these probabilities into account should give more weight to informative utterances. Formally, we use the geometric mean pGM(g|c) of a gender g for a character c given the individual utterances u(c) for the character:

In Figure 1, utterance 1 is low probability male, utterances 2 and 4 are high probability male, and utterance 3 is low probability female, leading to a character-level aggregated classification of male.

The topic of making machine learning models explainable has seen considerable attention in recent years, from a variety of motivations (Riedl 2019). Two major families (Murdoch et al. 2019) are model-based approaches – which build on the inherent interpretability of small models – and post-hoc methods – which analyze previously trained models to explain their predictions. In turn, post-hoc methods can be grouped into global and local methods. Global methods analyze the entirety of a model’s predictions over some dataset, while local models explain individual predictions (Atanasova et al. 2020). In our study, we focus on local methods because we are interested in looking at individual predictions. A prominent local approach for the attention-based models ubiquitous in NLP is the analysis of attention weights; however, it has been shown that these models are not guaranteed to faithfully attribute the model’s prediction to its input features (Jain and Wallace 2019). Therefore, we adopt the so-called integrated gradient approach by Sundararajan et al. (2017). Applied to a binary classification task with classes A and B, this approach explains every prediction as a sum of weights (positive, i.e, supporting class A, or negative, i.e., supporting class B) over the input features. In our case, every word in the input text is assigned a weight that indicates how strongly it supports (or contradicts) the assignment of a particular gender to the text.

Figure 2 shows some example sentences from a Calderón drama, where orange color indicates words that are typical for female speech and blue color for male speech. Again, saturation corresponds to the strength of the cue. Note that these attributions can combine morphological and semantic considerations: The most strongly female token, “desesperada” (desperate), is more likely to be uttered by a woman both due to its female suffix -a, and by topical association.

A final complication in our use of explanation methods to mine a classifier for information about an author’s linguistic patterns is that the features presented to the classifier need to be informative for the research question. This is not guaranteed, since a classifier can base its decision on features that are uninteresting from a theoretical perspective. For example, person names are informative for gender classification since men and women interact more within their genders than across. At the same time, person names are arguably uninteresting with respect to gender portrayal. In Calderón’s tragedy Un castigo en tres venganzas, for example, only the main character Federico ever addresses the character Becoquín – meaning when the token Becoquín appears, it is with certainty spoken by the character Federico and thus a perfect cue for male gender. More generally, the problem is the Zipf distribution of natural language, as a consequence of which there is a large number of infrequent words that occur either only in male or in female speech and could therefore be identified by the classifier as highly informative for the respective class.

This can be seen as an instance of overfitting (Wolfe and Caliskan 2021). The problem is also well known in stylometry (Evert et al. 2017; Šeļa et al. 2024). It is usually addressed by requiring words to meet certain frequency conditions, such as top-n words, mid-frequent words (Caliskan et al. 2022) or words that occur in a minimum number of documents. Another strategy is to mask particularly predictive tokens (Tang et al. 2022), thus making them unavailable to the classifier and forcing it to focus on more general cues.

Impact of Input Levels. Table 1 shows the main results for gender classification at the three input levels without any aggregation. The model performs best at the character level (F1 = 0.83), followed by the scene (F1 = 0.70) and utterance (F1 = 0.68) levels. This is not very surprising, given that the character level provides the model with the longest, and thus most informative, input texts. It is interesting, though, that the performance of the model drops by 15 points F1 score between the character and the utterance level while there is hardly any difference between the scene level and the utterance level. The baseline results change hardly between levels, indicating that this effect does not reflect a change in the difficulty of the task. We therefore draw as our first conclusion that reliable gender identification requires more information than is typically available from a single utterance or even scene.

To explore the failure modes, Table 2 shows F1 scores for our gender classifiers at the scene and utterance levels. We split these numbers into four quartiles of characters in terms of numbers of words spoken. The results again demonstrate a clear improvement for longer texts of 15-20 points F1 score between the first and the forth quartile. Evidently, many short utterances are not informative enough to permit reliable gender determination of the speaker.

Impact of Aggregation Method. Next, Table 3 presents the performance of the models when classification is first carried out at the utterance or scene level and then aggregated to the character level. The numbers in Table 3 are comparable to the ‘Character per drama’ row of Table 1.

Comparing the two aggregation methods, we observe that the majority vote performs rather badly, leading to a substantial loss in performance compared to the predictions without aggregation in Table 1. In contrast, the combination by geometric mean performs well, and aggregated predictions at the scene level almost reach the performance of the model that considers the complete text of a character.

This pattern meshes well with our interpretation above: When we use the probability estimates of the model, uninformative utterances have less impact on the overall prediction, and performance improves. All in all, we believe that the aggregated scene-level model is the most interesting among the ones we consider, since it is almost as good as the character-level model when predictions at the character level are evaluated, but also provides a separate prediction for each character in each scene, making it possible to pick up changes in gender roles over the course of a play – exactly the phenomenon that we are going to investigate below in Experiment 2 (subsection 4.2). Thus, our second analysis yields a somewhat more nuanced picture: Gender classification requires a fair amount of text, but the effect can be mitigated by informed aggregation.

Predictions for Male vs. Female Characters. Table 4 shows the results of the non-baseline models broken down by gender; best results for each metric and each gender are boldfaced. The results for male characters are generally better than for female characters, which is expected, given the higher number of male characters. That said, there is a striking difference in the effect of the aggregation strategy between genders: While results for male characters show only a rather small change of 5 points between the character and utterance levels, results for female drastically improve at the higher input levels (+.4 at the scene level and +16 at the drama level). When taking into account just the majority label for each character, the performance drops, especially for the female characters, at both the scene and utterance levels. When calculating the geometric mean, the performance at the scene and utterance level for both male and female characters increases again, in particular for female characters. Again, this goes well with our general observation that some sentences are highly gendered while others are uninformative as far as the model is concerned. Thus, the geometric mean-based aggregation can be seen as a strategy to reduce gender bias, according to its definition as performance difference between genders (Rudinger et al. 2018).

To further validate this interpretation, we examined both the 10 correct and 10 incorrect characters with the highest confidence according to the model. We initially assumed that the gender annotations found in the DraCor corpus were gold standard. However, through this examination we identified several annotation mistakes in the corpus. Among the 10 most confident ‘incorrect’ male predictions, were two female characters incorrectly annotated as male (Palas and Glauca). Likewise, among the 10 most confident ‘incorrect’ female annotations, we found an erroneously annotated character (Simón).

Attribution Model. From the quantitative model evaluation, it is clear that the model is able to predict the gender of characters in Calderón’s plays and even identify some annotation mistakes. However, it is not clear yet to what extent the model drew on features of the input that actually have to do with gender portrayal, despite our efforts to remove confounder features (see the discussion on lexical overfitting in section 2). By using an attribution model, we now analyze concretely on the basis of which tokens the model makes its classification decisions. In the following analysis, we list BETO tokens together with their integrated gradient scores, where negative scores indicate a tendency towards female speech, and positive scores a tendency towards male speech. Tokens with values closer to 0 are gender neutral, according to the model.

In order to determine if there were clear themes in the tokens that the model judged as being more masculine or more feminine, we first examined the averaged attribution score of each token, calculated as the average attribution score of the token over all sentences in the validation set in which it occurs. Table 5 lists the 20 tokens deemed to be the most feminine and masculine, respectively.5

We see several patterns in the attribution scores. Included in the most feminine tokens are several word endings or words with feminine endings (cansada, desesperada, contenta, celosa, todas, ##ísimas, ##acas). These morphologically grounded indicators of female characters indicate that in Calderón’s plays, female characters either refer to themselves or other women frequently. At the semantic level, we see that some of the most feminine words have some semantic connection to interpersonal relationships between characters (amigas, beso, discursos, cuerpo, celosa). This indicates that, broadly throughout Calderón’s works, female characters are more likely to discuss themes of love and relationships than male characters. Others among these words (cansada, desesperada, contenta, celosa) also demonstrate a depiction of women as emotional – the women in these plays are on the one hand able to discuss their emotional inner worlds, but on the other hand depicted as driven or controlled by their emotions (Drouet 2017).

This finding ties in with the results in Šeļa et al. (2024), who show that female characters across different languages and time periods are more likely to reference their emotions than male characters. Here we can see that Calderón’s work somewhat conforms to these norms, but we will later see some evidence of the subversion of this trope in subsection 4.2.

The male list contains several titles (Majestad, Emperador, Duque, Gobernador), which indicates the same pattern of more within-gender than across-gender interactions for male characters. Furthermore, these words indicate the importance of status in male-male interactions, arguably closely related to the topic of honor which was established as important across many works by Calderonian scholarship.6 Corresponding to the female indicators for conversations about relationships, the male list shows several words semantically related to combat (tropas, infantería, torneo, soldado), apparently the prototypical male topic. As previously mentioned, honor is of primary importance in Calderón’s works. In addition to the fact that war is a typically male endeavor, in the siglo de oro, male characters frequently use violence as a means to gain or restore honor (Honig 1972). It is therefore unsurprising that these themes feature prominently as ‘masculine’ words.

Curiously, nosotras is the token most attributed as male, however, it is not spoken by a male character in the corpus. It appears in the validation set 8 times, and of the 8 characters who do speak this token, 7 of them are correctly classified as female characters. Upon further investigation, it was found that nosotras appears in the training corpus, spoken by a female character that is mislabeled as male. Its listing as a female token is therefore the result of an annotation mistake, beyond the scope of our current study.

We also analyzed a list of top-40 male and female tokens with regard to their morphosyntactic properties, given that word classes or word types can also be used for stylistic gender portrayal (Caliskan et al. 2022). We found that the most female tokens included significantly more word fragments (15), compared to the male tokens (10). Conversely, the male tokens included more nouns (24) compared to the female tokens (14). The female tokens also included slightly more adjectives (5) compared to male (3). Both used an equal number of verbs. In conclusion, there do not appear to be major differences at the level of word classes – if anything, female characters tend to use less frequent words that tend to get split up into subword tokens.

Overall, this inspection of the features indicates that Calderón’s gender portrayal – within the space of features considered by the model – has two main components: (a) at the interaction level, women and men both prefer interlocutors of the same gender; (b) at the content level, women are more concerned with the domestic sphere and men the military sphere. There do not seem to be major formal linguistic differences between the two genders.

Note that we only inspected the top tokens associated with either gender, and therefore this analysis cannot make any strong claim as to completeness. It is indeed likely that other topics arise from a further analysis of such tokens. Furthermore, the masking procedure we carried out during processing can be repeated after this type of analysis: If, for example, the analyst decides that tokens indicating patterns of interaction are not interesting in terms of gender portrayal, these tokens can be masked, and the model training repeated. The result is a model which should have the ‘next best’ set of features among its top-ranking gender cues.

Cross-dressing characters are characters that adopt some traits of another gender during some scenes, dressing and speaking as a member of another gender. Such characters are a particularly interesting use case for our classifier-based gender portrayal method.

We obtained a list of five cross-dressing characters in four of Calderón’s plays, listed in Table 6, from the Role Call database.7 Since we are not aware of other sources for instances of cross-dressing in Calderón’s work, we treat this list as ground truth, although the possibility remains that the list is not exhaustive. All five characters are female characters cross-dressing as men: Rosaura, Eugenia, Semíramis, Claridiana, and Lindabridis. Each of these female characters subvert traditional roles by embodying manly traits such as intellectual aspirations, interest in war and hunting, physical strength, and resistance to marriage (Araico 2017). Information about in which scenes the characters are cross-dressing are taken both from the Role Call database and from looking at stage directions. Unfortunately, there are many inconsistencies, and the cross-dressing is not always explicitly stated in the stage directions. In the cases where the stage direction indicates cross-dressing, we use this information, otherwise we revert to the Role Call list.

For the ecological validity of this experiment, we ensure that all cross-dressing characters are part of the validation set – that is, the gender classification model is not trained on any cross-dressing characters. As motivated above, we use our trained model that aggregates predictions at the scene level. We do not carry out any specific prediction for these characters, but simply analyze the predictions of the model, as in Experiment 1, but now at the level of specific characters. We structure our analysis along two hypotheses.

Hypothesis 1: Lower Confidence for Cross-Dressers. If the model picks up on gender differences used strategically by Calderón, the predictions for the cross-dressing characters should be less confident about their gender than for other characters. Figure 3 shows the model’s overall confidence at the character level for all female characters from the four relevant dramas (Table 6), with the five cross-dressers marked by green boxes. Compared to other female characters that speak the same number of words, the model predicted cross-dressing characters to be female with less confidence (mean = 0.78 compared to mean = 0.89). Only four female characters are predicted with comparably low confidence (Clariana, Leonor, Celia, and Aurora), all speaking comparatively little. This pattern makes sense, given that cross-dressing characters mix female-like and male-like speech, providing conflicting evidence to the model.

Additionally, the model struggles with the overall classification of the cross-dressed characters more than with non-cross dressing ones (the accuracy is 4/5=0.80 compared to 19/23 =0.83). Again, this is expected, given that we would expect a data-driven model to classify a cross-dressing character either way, depending on the extent of their cross-dressing. Indeed, the ‘misclassified’ character is Claridiana, who cross-dresses in the whole play (see below).

Hypothesis 2: Scene-By-Scene Predictions. If the model does a good job of picking up gender cues in a fine-grained manner, it should be able to tell in which scene a character is cross-dressed. We test this hypothesis on the four relevant dramas, and Figure 4 shows the results: The cross-dressing scenes (according to the Role Call database, and in some cases identified by us in the few instances where stage directions indicate cross-dressing) are shown by a blue shading, and the model predictions are indicated on the y axis: the probability that a character was predicted to be female. Along the x axis are all the scenes in which a character speaks (ex. 1:2 on the axis represents Act 1, Scene 2). We find that, with a proportion higher than chance, cross-dressing characters were predicted as male for the scenes in which they were cross-dressing – the model is able to pick up on instances in which these female characters are behaving in a stereotypically masculine fashion.

The drama La hija del aire II focuses heavily on themes of governance and war as well as the mythical figure Semíramis, the queen of Asiria. Semíramis is described as being competent, a brave warlord, cruel and treacherous, but also evoking feminine sensuality. In the act three, after the death of her son, Semíramis disguises herself as her son in order to assume his place on the throne. In all of the scenes that Semíramis dresses as her son, the model predicts her as male, whereas through the rest of the play the model mostly predicts her as female.

In the story of El Castillo de Lindabridis, Lindabridis is a princess who travels the world in a flying castle looking for a husband. In this work, Claridiana is a female character who disguises as a male throughout the play in order to win the hand of Lindabridis. Claridiana engages in the trials of courage and combat required to marry Lindabridis. Claridiana is predicted by the model to be male in 14 out of 21 scenes in which she occurs. Lindabridis also disguises herself as a man in the third act of the play and in 6 out of 11 scenes of the third act she is predicted as male.

Finally, the model does a good job in La vida es sueño, in which Rosaura disguises herself as a man in the first act of the play in an attempt to seek revenge on a man who took her honor. The cross-dressing is clearly picked up by the model because she is predicted as male in the majority of the first act (4/5 scenes) and mostly as female in the rest of the acts (8/11 scenes).

From these cases, we can see that not only is the the model able to predict the gender of characters with high accuracy, it is also able to detect specific scenes where female characters are cross-dressing. Therefore, we can see that the social context of the characters has a large influence on the predictions of the model. Still, a more fine-grained analysis of the individual plays would help contextualize the model’s mispredictions at the scene level.

In any case, the model is still able to predict many of these cross-dressing characters as women, but with lower confidence, indicating the complexity of the characters – they are women and embody traditional female traits, while using cross-dressing as a means to transcend their social role as women and accomplish tasks in the public social sphere, wield power, and take agency of their own stories (Woods 2013).