Investigating genre labels of individual texts

The experiment

In this experiment, I will use cosine similarity scores to help detect texts with potentially wrong genre labels. Remember that the genre labels I have attached to individual texts (novels) come from external sources, such as Wikipedia and goodreads. So the question is whether our CS scores based on linguistic features of the texts support the given labels.

As in my earlier post, CBF genres: the big picture, I will start by comparing all texts with each other, apart from comparing a text with itself and with texts by the same author, and output their CS score. Note by the way, that the total number of texts in CBF has increased from 1,719 to 1,745.

Method

There are probably a multitude of ways by which we can measure linguistically how strongly a novel conforms to the conventions of a particular genre, e.g. theme, topic, style, etc.

Our previous posts have showed that comparing two modern works of fiction using the CS score seldom results in a high score, indicating much overlap in lexicon, parts of speech and semantic content. The exemption is if you compare novels by the same author. This means that most of the scores I get are below 0.3, which is considered a low CS score, i.e. the compared novels do not share many features, whether they are words, lemmas, parts of speech or semantic tags.

On the other hand, we saw in the previous post, that most texts in the CBF corpus will be more similar to texts with the same genre label than with texts with other labels, with some notable exceptions. We have also seen, at least for some novels of the crime genre, that even if a particular (Crime) text shares more features with one (or two) text(s) from a different genre, it will more often than not also share quite a few features with texts of the same genre, and these latter scores will be among the, e.g., five top CS scores for that text when compared with all other texts in the corpus.

So, what I will do in this experiment is to sort the CS scores and output all texts where a text's genre label is different from the genre labels of the top five texts that get the highest CS scores. A case in point is text fp201410L3 by Valentine Williams with the genre label Crime, where the top five CS scores are texts with the labels Romance (four) and Adventure (one). Interestingly, the five texts with the lowest CS scores for this text is Crime. Without looking at the text, one conclusion could be that this text has been mis-labelled. I could have done this kind of comparison for words, lemmas, parts of speech and semantic tags, but will limit the experiment to using lemmas.

Results

The number of texts where none of the five top comparison scores shares the genre label with the text they are compared with varies a lot between the genres. The reasons for this can be several, e.g. lack of enough (prototypical) texts of a genre, that the genre has changed or disappeared over time or that the genre is not very homogenous to start with. Table 1 shows the number of texts within each genre where the top five comparison scores do not correspond to the genre of the source or original text.

Admittedly, the numbers are small for some of the genres, but it would seem that Adventure, Humour, Spy and War are perhaps not inherently coherent enough, in this corpus at least, to warrant separate labels. The question is, if we were to relabel the texts of these genres, which of the other labels should we choose? One approach could be to attach the label out of the top five which the source text has been found to match most times, especially since the CS scores do not vary a lot between the top 5 matches (average: 0.38, min. 0.27, max. 0.57). 

For Adventure this is Historical, which comes out on top with with 12 of the 28 matches, followed by General (7), Crime (5), Romance (2) and War (2). For Humour the numbers are General (16), Crime (8) and Romance (3). For Spy they are Crime (18), General (4), Adventure (1), Romance (1) and War (1). And, finally, for War they are General (6), Crime (3), Historical (3) and Adventure (1). Based on these numbers, a relabelling along these lines seems sensible:

• Adventure --> Historical
• Humour --> General
• Spy --> Crime
• War --> General

Doing away with Adventure, Humour, Spy and War would also facilitate further comparisons of the texts, since we will get more robust genre categories and perhaps, for Adventure at least, not having to suspect that a genre category is based on other criteria than theme, topic, etc,. e.g. date of publication. The downside, on the other hand, is that operating with fewer, but bigger categories may hide subtle differences between (groups of) texts within one genre.