Are Some Books Better Than Others? An Analysis of Reader Responses

1. Introduction

How much of a novel’s merit is debatable? To what degree is good fiction in the eye of the beholder? Newspaper articles and awards committees often provide strong opinions, thereby steering the economic fate of authors and determining what the public considers to be good literature (Chuey et al. 2024). In this paper, we set out to empirically disentangle the relative contributions of book and reader characteristics in determining online book reviews.

2. Subjectivity in Book Ratings

Whether discussing architecture, music, or written works, people have always debated whether beauty comes down to personal taste or is, at least sometimes, undeniable (Ball 2004; Gessey-Jones et al. 2020; Hume [1757] 2017; Hyman 2002; Pelowski et al. 2017).

In the world of fiction writing, most creators, editors, and publishing professionals agree that different books appeal to different readers (Alharthi et al. 2018; Baumard et al. 2022; Waples 1931). As a clear sign of this, people’s literary taste changes over their lifespan; the very same reader might love a book as a child but then loathe it a few years later (Reuter 2007). Some books might entertain millions of adoring fans, while being snubbed by professional critics (Bizzoni et al. 2023). Once celebrated books might lose their appeal to modern audiences (Shaffi 2023). All this suggests that a reader’s perspective matters when assessing book quality. But how much?

Bizzoni and colleagues discuss whether mild perspectivism can provide an adequate description of reading enjoyment, where books do vary in their average enjoyability, but differentiable reader groups have unique criteria for books. As examples of reader groups, the authors list professional critics, lay readers, and demographic groups, which vary in their reading appreciation (Bizzoni et al. 2022). In fact, readers themselves report enjoying different literary genres (Herald and Stavole-Carter 2019) and such self-ascribed preferences affect how they search for new books (Mikkonen and Vakkari 2017), which reading communities they join (Kraaykamp and Dijkstra 1999; Thelwall and Kousha 2017), and which recommendations they receive by librarians and literature websites (Benkoussas et al. 2014; Thelwall and Kousha 2017).

The biggest divide in reading enjoyment lies between people claiming that they hate books in general and people who make sweeping declarations to love books (Greaney and Hegarty 1987). Long-term reading enthusiasts often connect books to various positive experiences throughout their life from bonding with parents to escaping their sorrows during difficult times (Carlsen and Sherrill 1988). They appreciate books as stimulants of their imagination, and often form habits around reading (Strommen and Mates 2004). Conversely, book skeptics tend to reject books for reasons of anticipated boredom or exhaustion (Gordon and Lu 2021), or because they genuinely do not think they have the skills to read a book (Unrau et al. 2018). Teachers complain that many students lack patience for books or have been poached by fast-paced, visual media (Poletti et al. 2016). Non-surprisingly, a person’s overall attitude towards books, as well as their self-perception as a reader, primes their subsequent enjoyment of books (Adelson et al. 2019; Ramirez et al. 2019). In sum, there is widespread agreement that a book’s perceived quality depends on characteristics of the reader, at least to some degree.

3. Objectivity in Book Ratings

While the reader’s perspective matters, most people also believe that some texts are truly better than others (Croxson et al. 2021). In fact, books are regularly bestowed with absolutist scores, reviews, and awards that celebrate the ‘best’ literature (National Book Foundation 2024). There are instructional handbooks on how to write better (White and Strunk 2023), lists of common writing mistakes (Mittelmark and Newman 2009), and detailed how-to-guides by famous authors (King 2000).

Most people would likely agree that a book filled with (unintentional) grammatical errors, logical inconsistencies, and written in a challenging font has some avoidable shortcomings. People are so certain that these things are mistakes that students are taught to avoid them in their writing. Admittedly, professional authors can eradicate such superficial errors quite easily; however, the mere observation that one can get better at writing suggests that authors might still differ on the latent dimension of ‘writing skill’. For instance, other common reader complaints, like unlikable or flat characters, are much more abstract and challenging to avoid, and thus more difficult to master than grammatical rule following (Balée 2006; Walsh and Antoniak 2021). An analysis by Kaufman and Kaufman suggests that authors often need more than ten years of professional experience to produce their best work (S. B. Kaufman and J. C. Kaufman 2007).

If authors can become better at writing, this entails that books can differ in their quality, provided that an adherence to writing conventions translates to increased appreciation by the readers. Tankard and Hendrickson tested whether one of the most famous writing tips (Show, do not tell!) actually has a positive influence on reading enjoyment (Tankard and Hendrickson 1996). They found that the descriptive show-versions of sentences (e.g., “Suddenly I awoke in a drenching sweat, my heart racing”) were indeed perceived as more interesting and engaging than the abstract tell-versions (“Suddenly I awoke, frightened”). However, not all writing conventions correlate with book popularity as Boyd and colleagues found no benefit of choosing any narrative structure over another in a sample of 60,000 works of fiction (Boyd et al. 2020).

In sum, it is a common belief, backed by some empirical evidence, that a story can be written well or poorly, meaning that some books might truly be better than others. We are left with two questions. First, to what degree is a book’s perceived quality due to the book, versus the reader? And second, how could one measure a book’s quality?

4. How to Measure Book Quality

It turns out that the aforementioned questions are two sides of the same coin. Both the nature and the measurement of book quality pose the problem of which reviewer to believe. One field that battles this problem is computational literary studies, where machine learning algorithms are employed to predict a book’s success. (Notice the cliched ‘coin’ metaphor in the paragraph’s first sentence. Did it hamper your reading enjoyment, as it is unoriginal, or did you find it useful, as it is succinct?)

While some researchers focus on predicting book sales or download counts (Ganjigunte Ashok et al. 2013; Otten et al. 2019; Silva et al. 2024; Sujo et al. 2021; Wang et al. 2019; Yucesoy et al. 2018), others explicitly forecast canonicity, award wins, or review scores (Barré et al. 2023; Maharjan et al. 2017; Pascake Moreira et al. 2023; Pascale Moreira and Bizzoni 2023). Such prediction models have the power to steer publishers’ investments and therefore determine what the public reads and which authors become successful. However, can we assume that the chosen prediction targets – specifically reader reviews and committee scores – are reliable measures of a book’s quality?

Bizzoni et al. (2022) describe two opposing views in this regard: strong perspectivism and weak perspectivism. The former denotes that even a single review score constitutes an accurate measurement of a book’s quality, but might not generalize to other readers. Thus, a book’s quality should only be measured and predicted for the individual reader, and never in absolute terms. Weak perspectivism, on the other hand, argues that book quality can be measured in absolute terms, but might require averaging across many reviews by different people. It postulates that individual review scores serve as noisy indicators of a book’s latent quality. However, how much idiosyncratic noise overlays the latent book quality remains unknown.

Researchers predicting aesthetics ratings in other fields have quantified the amount of inter-individual variation to distinguish it from other forms of statistical ‘noise’. For instance, Hönekopp (2006) found that judgments of facial beauty are about equally determined by face-characteristics and rater-tendencies. Hehman et al. (2017) extended this finding by showing that some face ratings do depend on the observed face (e.g., perceived happiness) whereas other traits are almost entirely due to the beholder (e.g., perceived creativity). Human-generated art like paintings, architecture, and textiles elicit more person-specific aesthetics ratings than faces and nature scenes (Vessel et al. 2018), a trend that is more pronounced for abstract art than representational art (Leder et al. 2016; Schepman et al. 2015).

When it comes to text evaluations, there is evidence for both subjectivity and objectivity. The quality of news domains, for instance, appears to be rated with high agreement among experts (Lin et al. 2023), whereas reviewers of scientific manuscripts often disagree in their evaluations (Siler et al. 2015).

The appreciation of textual humor appears to be largely subjective (Rosenbusch et al. 2022) and, as textual humor is a form of creative writing, we decided to pre-register our numerical hypotheses – about the relative importance of books and readers – in direct accordance with the confidence intervals for written jokes (cf. Rosenbusch et al. 2022; study 2):

Relative-importance-hypothesis

Differences between raters will account for more than three times as much rating variance as differences between books.

Book-importance-hypothesis

Differences between books will account for five to nine percent of all rating variance.

Rater-importance-hypothesis

Differences between raters will account for thirty to thirty-eight percent of all rating variance.

Next to broadening our knowledge of subjectivity in text evaluations, the presented analyses will provide answers to practical questions like: Should one use book reviews for purchasing decisions? How many reviewers are needed to assess a book’s quality and make fair comparisons between books? And which pieces of information are needed to predict a book’s reception by the public?

5. Method and Materials

Pre-registration, code, and data can be found in section 8 and section 9. We analyze book reviews from one of the largest literary websites: goodreads.com (cf., Walsh and Antoniak 2021). Goodreads users rate books by clicking on a star scale, reaching from one to five stars, or by writing a free text review.

To collect the data, a Python script iterated through random numbers (numpy.random.randint; seeded) and inserted them into the website’s url, stopping after 300,000 numbers. It is unknown how Goodreads assigns URLs and we therefore decided that uniform sampling posed the lowest risk of sampling bias. Given that webscraping is generally disfavored/blocked by website hosts, and given that Goodreads has shut down its API for programatically retrieving book reviews, it is possible that our data collection might not be reproducible in the future. In this case, researchers will be able to reanalyze our numerical, anonymous data. Private accounts, and reviews that were not posted publicly for the entire reading community, were not (and cannot be) accessed. From each random user, up to twenty random reviews were collected. Users with zero or one review were also discarded. No identifying data were collected. The final dataset consists of 566,121 ratings, from 67,012 participants, and across 49,674 books. Users annotated the book corpus with 875 unique genres ranging from very general (‘fiction’) to very specific tags (‘World War II’). In line with our preregistration, no genre subsampling or weighting was performed.

During the analyses, we disentangle two sources of variance, the book and the reviewer, through the use of random effects in multilevel models and the ICC value associated with each source of variance. Thus, we operationalize differences between books and differences between readers as inter-book and inter-reader variance in book ratings, respectively. We further show probabilities of book ratings conditional on other ratings to assess rating consistency across readers and books. Confidence intervals are computed by bootstrap sampling each analysis with ten repetitions and noting the highest and lowest values. Note that we pre-registered 100 rather than ten repetitions, which were unneeded given the large sample size.

6. Results

A multilevel model with random intercepts per book and reader (Bates et al. 2015, 2024) revealed that differences between books account for 3.53% [3.42–3.68] of the rating variance on Goodreads, whereas differences between raters accounted for 30.28% [30.06–30.51]. Reader differences were 8.58 times [8.2–8.87] more impactful than differences between books. Figure 1 illustrates the relative benefit of knowing the rater vs. knowing the book when trying to predict rating scores. The relatively high consistency of ratings from a given reader, compared to the lack of consistency for ratings of the same book, is also highlighted by an extended simulation where we sampled an increasing number of raters and averaged their score to approximate either a single rater’s score, or the book’s average score in our complete dataset (cf. Figure 2).

Figure 1: Top left: conditional probability of a book rating, given another reader’s rating of the same book (If Rater A gave a book one star, there is a 50% chance that Rater B will give it five stars). Bottom left: conditional probability of a book rating, given the same reader’s rating of another book. Top right: consistency between average book ratings (Ns=3), all made about the same book. Bottom right: consistency between average book ratings (Ns=3), all made by the same reader. Takeaway: Two raters rarely agree (tops), but individual raters have consistent tendencies (bottoms).

Figure 2: Left: Pearson correlation of sample ratings with the rating of a single rater. Right: correlation of sample ratings with books’ overall ratings in our dataset. Notice that these correlation coefficients lie far below conventions for reliable quality assessments (Montgomery et al. 2002), despite being inflated by the rating samples forming part of the ‘All Ratings’ average.

6.3 Analysis of Review Topics

As a final analysis, we examined whether readers bring up similar issues when reviewing the same book. It might be that readers disagree widely in their overall book ratings, while still perceiving a similar set of strengths and weaknesses per book (of which the importance could vary idiosyncratically). We used a large language model (GPT-4o) to annotate written reviews (N = 26,699) for fiction books regarding the mentioning of feeling bored, addicted, or confused by the book contents, as well as the mentioning of characters, and the author’s unique style or skill in writing. We chose these concepts as they relate to common facets of enjoyment and broad book distinctions (cf., page-turning thrillers vs. literary/character-driven fiction). We also annotated the presence of a book summary in the review as a control variable, as we did not expect books to differ much in whether they elicit summaries from readers. A Python script iterated through the reviews and was given a system prompt to “answer with numerical annotations” pertaining to the themes above (for the full prompt, see finetuning.qmd in supplementary materials). Subsequently, we went through the annotations with an R loop, displaying the LLM responses, and manually scoring the number of mistakes per review between 0 and 6 (based on the number of possible themes; see examine_gpt_performance.R and annotation_finetuning.csv in supplementary materials). GPT-4o achieved an annotation accuracy of 98.6% [97.74-99.11] on 200 human-annotated book reviews.

For our analysis, we drew pairs of reviews either targeting the same book or stemming from the same reviewer. As shown in Figure 3, the presence of specific review attributes was better predicted by other reviews of the same reviewer, compared to other reviews about the same book. Especially the writing of summaries seems to be highly characteristic of specific reviewers.

Figure 3: The X axis shows various topics that were brought up in book reviews. The Y axis shows how much more likely it is that a review mentions this topic if it was mentioned in a reference review, i.e., $\frac{P(\text{Topic}|\text{mentioned in reference})}{P(\text{Topic}|\text{not mentioned in reference})}$

For instance, the black triangular point on the left indicates that a review is twice as likely to mention that the reviewer felt addicted to the book, if the same reviewer made that claim about a different (randomly chosen) book. The gray circle below the triangle indicates that a review is about 1.6 times more likely to contain “feeling addicted” if another (randomly chosen) reviewer made that statement.

7. Discussion

How much value should you place on a stranger’s opinion when looking for an enjoyable read? Our analyses suggest that you should not place any at all. A single book rating will generally tell you much more about the reviewer’s average rating than the book’s – up to ten times more in our analyses.

While it is tempting to proclaim aesthetic value in absolute terms (‘must-reads’ vs. ‘trash-novels’), we find rater characteristics had a much stronger impact on book reviews than a book’s general appeal (here: its average rating). This is in line with previous work on the appreciation of jokes and abstract art (Leder et al. 2016; Rosenbusch et al. 2022). Across star ratings and written reviews, the impact of book characteristics (3-4% of rating variance) was even lower than our a-priori estimates (5-8%).

While genre fans also differed widely in their evaluations of genre books, reviewers with relatively many reviews and ratings achieved higher agreement (7-8%). That is to say that they managed to approximate the average book evaluation better than the average reader, which might be a homogenizing effect of expertise (i.e., a trained eye). However, even for reviews from regular Goodreads review writers, one should consider that their ratings remain virtually uncorrelated to the reading enjoyment of another individual reader. All in all, we conclude that one should not just apply mild but extreme perspectivism when talking about the enjoyability of books.

In addition to the numerical sentiments of written reviews (which likely lost some informative variance during binarization), we also found that specific complaints and compliments are more characteristic of the reader than the book. Thus, we speculate that readers do not just apply a different set of weights for consensual book criteria, but rather apply altogether different criteria (Annalyn et al. 2018; Kraxenberger et al. 2021). This does not mean that book ratings are ‘random’. People read books for different reasons and thus end up bestowing different evaluations and critiques (Kraxenberger et al. 2021). Thus, it is all the more noteworthy when reviewers do agree in their comments, as such an outlying degree of consensus hints at book contents that deviate so much from the norm that readers cannot help but converge in their reviews. For instance, today (30th of September 2024) the latest reviews of the book If on a Winter’s Night a Traveler almost all mention that it is confusing or at least challenging to read (which some readers report to enjoy), indicating that unique books lead to more homogenous review contents. Accordingly, we want to mention that online platforms can increase the informativeness of their comment section by implementing an upvote feature and increase the visibility of highly upvoted comments (as Goodreads does). Such response aggregations can boost the wisdom-of-the-crowd and facilitate faster and more reliable book insights from (shared) reader responses.

In the world of scientific peer review, it is well-known that manuscript evaluations are rarely consensual and often deviate from some ‘true’ metric (Montgomery et al. 2002; Siler et al. 2015). A popular method to strengthen the association between text contents and evaluation is the introduction of narrowly defined rubrics (Jonsson and Svingby 2007). For fiction writing, such rubrics could target criteria which are believed to predict reading enjoyment for a wide audience (e.g., “Does this book feature a protagonist with desirable characteristic X?” or “Is the adverb prevalence in this book below X%?”). While high-dimensional annotations of books can likely be generated by AI models (cf., our annotation of book reviews above, Bail 2024) it remains to be seen whether such rubric-style annotations can approximate a single person’s reading enjoyment, especially as reading preferences clearly vary across, for instance, people’s motivations and personality traits (Annalyn et al. 2018; Kraxenberger et al. 2021).

Viewing the current results, people certainly should not feel discouraged to read reviews or rate books themselves. Such activities are likely to enhance one’s reading appreciation by adding social relevance and offering new perspectives on books (Foasberg 2012).

While absolute book rankings can serve as indicators of prestige (see Feldkamp et al. (2024) for a review of quality metrics), they cannot predict a person’s enjoyment of a book. However, rather than eradicate them, they could simply be seen as descriptive of current audience trends or marketing success. Themed lists and book awards also offer value by directing readers towards specific book contents that they are interested in.

In that regard, it is also noteworthy that the lacking ‘accuracy’ of book evaluations does not necessarily preclude accurate book recommendations. When a reader’s relevant book preferences are measured accurately and matching books can be identified reliably, experts or recommendation engines can still function (as long as the measured preferences are sufficiently stable and highly predictive of reading enjoyment). However, general ratings and rankings can be largely discarded for recommendations and numerical scores of individual readers can be fully discarded, at least for professionally published books.

8. Data Availability

Data can be found here: https://github.com/hannesrosenbusch/are_some_books_better_than_others. It has been archived and is persistently available at: https://doi.org/10.17605/OSF.IO/46H7S.

9. Software Availability

Software can be found here: https://github.com/hannesrosenbusch/are_some_books_better_than_others. It has been archived and is persistently available at: https://doi.org/10.17605/OSF.IO/46H7S.

10. Author Contributions

Hannes Rosenbusch: Conceptualization, Investigation, Methodology, Software, Formal analysis, Validation, Visualization, Writing – original draft

Luke Korthals: Conceptualization, Writing – review & editing