Limericks and Computational Poetics: The Minimal Pairs Framework. Computational Challenges for Poetic Analysis and Synthesis

2.1 Language Models

The renaissance of neural network models (often marketed under the heading of ‘deep learning’) has greatly advanced expectations for a machine’s ability to perform machine reading and machine writing. Applied to language modeling (Goldberg 2017; Jurafsky and Martin 2021), these models present exciting opportunities for research in literary studies and computer-supported creative work.

Our work explores the power of the GPT-2, BERT, TransformerXL, and (causal) XLNet language models. Each is based on the computationally efficient ‘Transformer’ architecture (Vaswani et al. 2017), a basic mathematical model that, given some text, predicts with varying probabilities the surrounding text in any human-produced text instance. This is an encoding of each word in the vocabulary as a vector, which is effectively a list of numbers. This model depends on a range of numbers – parameters – that have been set according to the likelihoods of various text strings occurring in a large body of text, such as all the writing in Wikipedia. Pre-training is the algorithmic setting of these parameters based on the example text corpora.

The Transformer architecture derives from the better known ideas and architectures based on and inspired by “neural networks” (see e.g., Gurney (1997)), which are themselves loosely modeled on the ‘wet’ network of neurons in our own brains: connected collections of billions of simple cells (neurons) whose signaling patterns underlie the abilities of all animals to encode learning and learned behaviors. Early neural networks with a relatively small number of (mathematical) neurons were but one of a large number of basic ‘classifiers’, mathematical models for segmenting data and predictive algorithms.³ It was something of a surprise that when the model sizes were dramatically increased – going from tens of parameters to orders of magnitude larger – that neural networks showed great and in many ways unforeseen abilities to do data discrimination and prediction. Modern machine learning continues to ride the wave of the strength of these architectures and modern computing has enabled the ability to continuously fit more and more parameters in models of increasing size and complexity.

‘GPT’ stands for Generative Pre-trained Transformer. GPT, GPT-2, and now GPT-3 (Brown et al. 2020; Radford et al. 2019) are the three generations of a basic – GPT – architecture specially designed to produce human-level text. They come in ‘sizes’ (small, medium, large, etc.) and the successive generations are largely distinguished by the expansion of the number of parameters embedded in the models and the requisite sizes of the (pre-)training sets needed to tune these models – tens of billions in the case in GPT-2 and hundreds of billions in the case of GPT-3.

‘BERT’ stands for Bidirectional Encoder Representations from Transformer. The ‘bidirectional’ modifier reflects a model that looks at word sequences both backwards and forwards for training (Devlin et al. 2018).

‘TransformerXL’ (Dai et al. 2019) is a causal language model, meaning that unlike BERT it only uses preceding words to predict the next word. Its standout ability is the addition of a recurrence mechanism, which is a form of machine memory. Information from previous word sequence segments processed by the model (a fixed-length context window of words that the model uses to predict the next word) is carried over to the next segment, which theoretically allows TransformerXL to look farther behind in text to make a good prediction.

‘XLNet’ is an evolution of aforementioned TransformerXL model that uses a clever mathematical trick to approximate all possible orders of words in a sentence, where, instead of a fixed-order (left to right) context, the model is exposed to a randomly permuted sequence of all words that both precede and succeed the word we are predicting, while predicting the last word (or last few words) of this sequence. Since the new context includes tokens both from the left and right of the original context of the target word, the model is bidirectional like BERT. At the same time it can also be used for left-to-right decoding (i.e., generation), like GPT-2 (Yang et al. 2019).

2.2 BPoMP

The BPoMP method evaluates a language model by giving it a choice between a right and wrong instance, where the wrong instance is a ‘minimally’ doctored version of the correct – original – instance. For example, one might have Poem A, but then replace a single word in Poem A with a random word to create Poem B. The more capacious a language model, the more reliably it is able to distinguish between the original and the corruption. The design of minimal pairs to isolate a phenomenon of interest can be rather subtle.

There are three reasons to use BPoMP challenges when comparing language models’ ability to model poetry. First, the BPoMP challenges provide a useful ‘second opinion’ when considering model performance. Traditional measures of model ‘fit’ such as perplexity (see e.g., Chen et al. (1998)) are often unreliable or difficult to calculate in settings where the observed data is high dimensional. Performance on BPoMP challenges, by contrast, is easy to calculate and to interpret. Second, the BPoMP challenges can be used in settings where traditional evaluations such as perplexity are unavailable. As examples of this, BPoMP can be used to compare two language models which use different tokenization strategies and to compare a bidirectional language model with a traditional (‘causal’) language model. Third, using the BPoMP challenges to evaluate models can yield insights into the strengths and weaknesses of particular models. It is, for example, easy to imagine one language model performing better on BPoMP challenges involving rhyme but worse on all other challenges. Such a result may indicate that some component of the model is doing well at capturing regularities in language that relate to rhyme. If this component can be isolated, it could be borrowed by other models. While one might balk at leaving such evaluations of models to the machine, recent work supports the use of machine – rather than human – evaluation (Clark et al. 2021).

The preceding arguments in favor of the BPoMP challenges closely resemble those offered in favor of the BLiMP challenge set (Warstadt et al. 2020). Whereas BLiMP helps to bring into focus the strengths and weaknesses of language models’ ability to adequately model differences between grammatical and ungrammatical sentences, BPoMP helps researchers to characterize models’ capacities to capture differences between poetic and non-poetic language.

In Abdibayev et al. (2021b) a dataset of 10,000 minimal pairs of limerick/corrupted limericks was used in a task of ‘choosing’ between the two options to determine the original limerick. Transformer-based models were used. The minimal corruptions were: (1) shuffling two rhyming end-of-the-line words, (2) shuffling two rhyming lines, (3) replacing an end-of-the-line word by a non-rhyming synonym. While the models identified the original limerick at rates better than chance, there was a good deal of room for improvement. It is fair to say that the models have yet to demonstrate that they have developed an ear for poetry.

This work on detecting formal elements (poetic analysis) complements, but takes a fundamentally different approach from, useful, existing prosodic parsers and pedagogical tools – e.g., Prosodic and For Better For Verse.⁴ Whereas such tools are focused on discerning or teaching meter, our standardized package is concerned with investigating more fundamentally – via our adaptation of the minimal pairs method – what ‘poetic knowledge’ a language model possesses or can learn, and therefore has a more comprehensive scope which includes not only accentual patterning but other formal elements (rhyme, musical devices like alliteration, and so on).

2.3 Word Deletion

Word deletion is the focus of one of the new minimal pair benchmarks in this paper. In short, a language model is presented with a limerick and a near twin, corrupted by the removal of one, two, or three words. The model then ‘chooses’ between the two by calculating the likelihood of both poems. The text with the higher likelihood wins. This textual challenge probes the models’ ability to discern the semantic, syntactical, and grammatical integrity and form of the limerick genre. Word deletion should disturb these qualities and a language model should know this in the probabilistic sense in which it ‘knows’.

The word deletion tasks also recalls certain textual and literary practices. Presenting our models with an original text and a version of the same text with missing words mimics the longstanding problem of omissions in the historical transmission of texts. Missing words are an inevitability in the manuscript documentary record, and the scholarly practice of textual criticism has codified various kinds of deletions. For example, a manuscript might be missing words because of saut du même au même: When the same word is repeated in close proximity, the scribe “copies the text as far as its first occurrence … then looking back at the exemplar to see what he must copy next he inadvertently fixes his eye on the second occurrence of the word and proceeds from that point”. The result is that “the intervening words are omitted from his copy” (Reynolds and Wilson 1991). Missing words and other such common errors degrade the transcription, but are crucial for textual scholars in positing the relationships between different manuscripts (that is, the practice of stemmatics or stemmatology): Missing words can help to establish how related or unrelated a given manuscript is to other manuscript copies of the ‘original’ (the archetype) (Reynolds and Wilson 1991). The deletion-based BPoMP arguably resembles the first and most fundamental step in such textual scholarship – to identify the more ‘correct’ or more likely text, which is not missing words – and measures how well models can automatically carry out this task.

In a different literary context, the minimal pairs challenge also evokes erasure poetry. An erasure poem is a type of found poem, where an existing composition written by another is manipulated to create a new work through the blacking out or omission of some words or letters (American Poets 2022). Another way to look at the deletion task, then, is that our algorithm is taking an original limerick and creating an erasure poem out of it. This task raises all sorts of questions about the human and machine discernment of erasure poems. What exactly might distinguish ‘good’ erasure poetry from the randomized omission of a composition’s original words? Although we are assuming for our purposes that the erasure poem is the ‘incorrect’ choice, how might models learn to recognize legitimate or artistically compelling instances of erasure poems? The task also raises interesting ontological questions about the nature of found and manipulated poetry like erasure poems.

2.4 (Poor) Last Line Completion

Our second minimal pair experiment tests models on their ability to distinguish an existing limerick from a similar limerick in which the fifth line of the original limerick has been overwritten by a computer-generated fifth line. The first four lines of each poem in the pair are identical. This experiment involves, then, a stage where a natural language generation technique – whatever technique one uses – completes a four-line fragment of a limerick with a plausible fifth line that fulfills some of the requirements of the limerick form: primarily the a end rhyme (that rhymes with lines 1 and 2 in the aabba rhyme scheme) and a line length that also generally conforms to those of lines 1 and 2 (typically the longer lines of the limerick in terms of the number of words, compared to lines 3 and 4). (In our experiment, we did not require the synthetic fifth line to meet the rhythm requirement of three stresses and triplet meter.)

This challenge asks the language model being evaluated to discern the original limerick from the synthetic version. This is testing whether or not the model has some minimal sense of how limericks typically end. The machine-generated last line is not a typical line, but it is a good fake, so a model purporting to model narrative and/or semantics cannot ‘cheat’ by just looking to see if it has the right number of syllables and rhyme scheme. Thus, the last line completion minimal pairs also probe a language model’s ability to encode coherence in the limerick. Note that is possible that a language model performs well on word deletion and not on line completion or vice versa.

This test pushes the definition of a ‘minimal’ alteration to its limit in that we replace an entire line of a limerick. Yet this aspect of the experiment recalls the history of the limerick form. Although the origins of the limerick form are obscure, we do know that it was initially popularized in the nineteenth century by Edward Lear’s A Book of Nonsense (1846) and then reached another peak of popularity around the turn of the twentieth century. During this latter peak – called by one limerick historian “the great limerick boom” – several promotional competitions elicited the public to submit a final line for a limerick fragment, for a chance at winning substantial prize money (McInerney 2001). The most well-documented example in the limerick literature is a contest put on in 1907 by the cigarette company J. Samuda & Co.(Legman 1969). Offering a prize of 3£ per week for the rest of the winner’s life, Samuda’s contest asked the public to complete the following limerick fragment, which advertised the company’s product:

That the “Traylee’s” the Best Cigarette,
Is a tip that you cannot regret:
And in buying, I’ll mention
There’s a three-pound-a-week pension, …

J. Samuda & Co.’s advertisement proclaimed that the competition would identify “the greatest of all limericks”, but really, as the ad itself reveals, the competition was a gimmick aimed at promoting a new product, Traylee Cigarettes (see Figure 1). Hence contestants needed to mail in an order for the cigarettes in order to submit a fifth line for the limerick. Evidently, J. Samuda & Co. went on to hold several such promotional competitions involving limerick completion, alongside other similar competitions put on by others. In 1907 alone, there were more than 11 million postal orders for such limerick contests. The winning line for J. Samuda & Co.’s 1907 contest was, “Two good ‘lines’ – one you give, one you get”, punning on ‘line’ (the poetic line, the cigarette, and the financial life line) (McInerney 2001).

Figure 1

 J. Samuda & Co. advertisement.

In any case, what matters here is less the history of limerick contests per se than the fascinating resonances between the poetic challenges we have set for language models and such historical practices and precedents. These resonances – between manuscript omissions and deletion tasks, between popular poetry contests and the autoregressive generation of final limerick lines – suggest a rich direction for future media archaeology and historical poetics-inflected inquiries into the history of poetry (and texts generally) and the methodologies of contemporary computational literary studies.

2.5 Dataset

The limericks used for the research in this paper originated as a subset of the content from the website The Omnificent English Dictionary in Limerick Form (OEDILF). Established in 2004, it is an amateur, crowd-sourced project whose goal is to have at least one limerick for every meaning of every word found in the Oxford English Dictionary. User-submitted limericks are subject to approval by moderators, and, if approved, are published on the website.⁵ Among the benefits of OEDILF is that it comprises a large number of limericks which can be sorted according to different categories of metadata.⁶ On the website, the limericks are organized according to different categories: (a) authors, (b) topics. Last but not least, many printed anthologies of limericks highlight particularly misogynistic and/or racist limericks. While OEDILF has its share of ribald poems, it skews in such a way that it provides a large archive of poems from which we can create a good corpus.

We work from a subset of OEDILF originally gathered in Abdibayev et al. (2021b). Therein, two levels of filtering reduced the original 110,610 published limericks to a set of 65,000. The first level was based on simple structural criteria (limericks must have five lines and must use words – rather than symbols, like emojis or formulae). A next level kept only those limericks that verifiably – by machine – satisfied basic structural properties. Specifically, only limericks where all end-of-the-line words could be verified by machine as satisfying the rhyme scheme were kept. For our fifth line completion task, we only picked limericks whose end rhyme words for the first and second lines could be located within our rhyming dictionary. We then used this set for our beam-search algorithm to generate synthetic fifth lines (see subsection 3.4).

For the deleted words minimal pairs, we further filtered limericks to keep only those where at least three end-of-line words are found in Merriam-Webster’s Collegiate Thesaurus. This produces 34,699 limericks from which we sampled 10,000 limericks (to reduce computational burden). Later, we used this smaller set as a testing ground for the delayed beam-search task.

3.1 General Structure of all BPoMP Tasks

3.2 Formal Definition of BPoMP

Having established these concepts, we now can explain BPoMP’s structure in finer detail. A language model G takes as an input two sequences S and S^*, which correspond to tokenized limericks L and L^*, and processes each independently, in no particular order. L^* is a transformation of L, or more formally L^*=f(L), where f is a function that alters (‘minimally corrupts’) the original limerick L. The alterations are aimed at singling out particular linguistic phenomenon associated with limericks.

By processing L, we mean outputting G(S) which will provide a probability P for each token in S, the tokenized version of L. We denote all tokens in the vocabulary of the model as V.

Once we compute the probability for each token in both S and S^*, we compute the total probability of sequences S and S^*, via

where w_i is the word at a position i and w_<i are all words before within the limerick.

Or, rewritten for clarity:

$P(S)={\displaystyle \prod _{i=1}^{\left|S\right|}G}(w_1,\dots ,w_{i-1})$

G(w₁,…,w_i-1) is the language model’s estimate of an abstract ‘true’ probability of encountering token w_i given words that come before it.

Note that some models, such as BERT, violate this formulation by considering the ‘score’ of each word by looking both at preceding and succeeding words. There are workarounds to their more exotic ways of computing (Lau et al. 2020), what we can instead call the pseudo-likelihood of a word (as these formulations do not satisfy the formal properties of a probability distribution). We will not go into the details of this as it does not contribute to the understanding of our experiments. Bigger models (e.g., GPT-2-medium v. GPT-2) generally give estimates closer to the aforementioned ‘true’ probability.

In practice, we work with $\log P(S)={\displaystyle \sum _{i=1}^{\left|S\right|}\log }P(w_i)$ instead of P(S) as it simplifies the computation. However, for simplicity of exposition in the examples we will just use probability P. The beauty of this approach lies in its simplicity and universality across models: Unlike explicit classification, it requires no pre-training and no additional computational units on top of the existing model (adding which may introduce more discrepancies in the comparison between models). Moreover, it opens up possibility of studying how much ‘poetic knowledge’ a model can learn without being explicitly constructed to do so.

We now can delineate the general structure of any BPoMP task. Given a tokenized, human-written limerick S and its tokenized, automatically generated alteration S^*, we ask a model G to compute log P(S) and log P(S^*). Comparing the two numbers tells us whether the model finds the original or the alteration to be more likely. When it deems the original more likely it scores a point. Our overall metric is then a simple accuracy measure: Divide the total points the model scored by the total number of test examples used in the experiment. An example of a BPoMP test point is presented in Figure 2 and Figure 3.

Figure 2

Example of a BPoMP task, specifically, fifth line completion task. Figure 2 is an original, tokenized limerick with each token characterized by its own probability.

Figure 3

An altered limerick, where the fifth line was replaced by a machine-generated one. All probabilities were produced by GPT-2 medium. The colors of probabilities correspond to magnitude. The arrows represent what words were used by the model when predicting the outputted probability. For simplicity, we didn’t include all arrows from the second line onward (with the exception of the first word for explanatory purposes). Every model receives a start token at the beginning of every sequence, for which the probability is not computed (presumed to be 1).

3.3 New BPoMPs

We now present two new BPoMPS that further refine our understanding of capabilities of these large models to encode poetic concepts.

3.4 New BPoMP Challenge 1: Random Word Deletion Task

In this task, we alter the original limerick by deleting M words, for M ∈ {1,2,3} (each choice of M is a separate BPoMP task). By ‘word’ we mean any string that is at least two characters long, separated from other words by spaces. Whenever a word is deleted (whether one or several) its surrounding punctuation is preserved. As noted above, this task somewhat resembles the protocols of erasure poetry.

An important consideration is sequence length difference. In this case, the deletion will create a twin that is shorter in word length. This in turn will trivially increase its total probability because the summation of log-probabilities is equivalent to the multiplication of regular probabilities, which in itself is lower for longer sequences on average since we are multiplying numbers between 0 and 1. Thus, to correct for this effect we rescale the total probability of each sequence by length. In other words, we compare

against

This formulation will be used throughout the paper.

The results (presented in Table 1) show that most models have no difficulty distinguishing between original limericks and limericks whose semantics were altered by word removals. However, one model remains an outlier: XLNet performs very poorly compared to others. This can be attributed to the somewhat unnatural causal nature of the task.

A second important consideration is the possibility that the three models that perform well on this task (GPT-2, BERT, and TransformerXL) are merely picking up on the grammatical and syntactical conventions they have learned from their training data. In other words, the models are basically functioning as grammar checkers on sentences with missing words. That said, the training data for these models likely include poetic language, and so these models are not only detecting standard prose usage. Still, in order to begin to address this, in the second BPoMP task described below, we test how well models can detect a real limerick ending from a synthetic one – and, in some of those minimal pairs, the limericks exhibit comparable degrees of conformity to what be considered as general and standard English language usage.

Human test subjects were tasked with a slightly different task: to mark if limericks were altered or not. This is due to ease with which one can solve the machine task if presented with both limericks. Their results are uniformly high: Performance dips on three deleted words task, but that can largely be attributed to small sample size, and perhaps, the greater allowance judges made for what passes as a legitimate limerick.

Below is an example of a random deletion task with three words removed from a limerick:

3.5 Transformer Completion Task: Beam Search and Delayed Beam Search

In the Transformer completion task we compute the probabilities comparing an original limerick and its corruption obtained by replacing the fifth line with a line generated by another Transformer-based language model (a smaller version of GPT-2 – not one of the models that is being tested) given the first four lines. We ensure that the machine completion necessarily rhymes with the first two lines according to the limerick rhyming scheme. In human evaluation, participants picked out the machine-completed line in all 40 of the test pairs.

4.1 Rhyme Probability and Artistry

The second BPoMP challenge necessitated generating synthetic fifth lines, only a percentage of which had correct end rhymes to match lines 1 and 2 of the source limerick (given the aabba rhyme scheme). In future work, we hope to explore more minutely how language models fare in generating different kinds of rhyme words given a certain initial rhyme, and the broader implications of improbable or difficult rhyme words for poetic artistry.

4.2 Poetic Minimal Pairs Examples

The investigation into the poetic knowledge of language models approaches poeticity or literariness using a novel approach. We anticipate future work expanding the BPoMP framework to other kinds of poems beyond the limerick and to other poetic features. Below are a next ‘level’ of examples of minimal pairs. Put aside, for now is the issue of preparing such sets a necessary sub-step that surfaces its own interesting set of challenges in computational poetics.

Great streets of silence led away
To neighborhoods of pause –
Here was no notice – no dissent –
No universe – no laws.

Great streets of silence led away
To neighborhoods of pause –
Here was no notice – no resistance –
No universe – no laws.

The night is chill, the cloud is gray:
‘Tis a month before the month of May…

The night is chill, the cloud is gray:
‘Tis many months before the month of May…

…Made weak by time and fate, but strong in will
To strive, to seek, to find, and not to yield.

…Made weak by time and fate, but strong in will
To strive, to seek, to find, and not perish.

Full many a gem of purest ray serene,
           The dark unfathom’d caves of ocean bear:
Full many a flow’r is born to blush unseen,
           And waste its sweetness on the desert air.

Full many a gem of purest ray serene,
           The dark unfathom’d caves of ocean bear:
Full many a flow’r is born to blush unseen,
           And waste its sweetness on the desert sand.

There was an Old Man with a beard,
Who said, “It is just as I feared! –
Two Owls and a Hen, four Larks and a Wren,
Have all built their nests in my beard!”

There was an Old Man with a beard,
Who said, “It is just as I feared! –
Two Owls and a Hen, four Larks and a Crow,
Have all built their nests in my beard!”

Thou still unravished bride of quietness,
Thou foster-child of silence and slow time…

Thou still unravished bride of quietness,
Thou foster-child of muteness and slow time…

When to the sessions of sweet silent thought
I summon up remembrance of things past

When to the sessions of sweet silent thought
I conjure up remembrance of things past

That night when joy began
Our narrowest veins to flush,
We waited for the flash
Of morning’s levelled gun.

That night when joy began
Our narrowest veins to flush,
We waited for the blaze
Of morning’s levelled gun.

Oh, never have I seen a dog so bare!
Naked and pink, without a single hair…
Startled, the passersby draw back and stare.

Oh, never have I seen a dog so bare!
Naked and pink, without a single care…
Startled, the passersby draw back and stare.

Much Madness is divinest Sense –
To a discerning Eye –
Much Sense – the starkest Madness…

Much Madness is divinest Sense –
To a discerning Eye –
Much Nonsense – the starkest Madness…