CBF and cosine similarity: the crime genre

Hypothesis: Crime fiction is a homogenous literary genre

If we compare selected crime novels of the 20th century with other literary genres, including itself (crime), can we expect the cosine similarity to always be higher comparing crime with crime than when comparing crime to e.g. general fiction, romance or spy fiction?

Caveats

An important point to have in mind when we look at the results below is the fact that crime fiction encompasses several sub-genres: detective fiction, police novels, cozy crime, (legal) thrillers(?), various kind of the noir sub-genre (Scandi-, Tartan and domestic noir) and more or less related genres, such as horror stories and spy novels. Moreover, all of these can be set in e.g. a historical period "blending crime fiction with historical detail" (ref. Copilot prompt: "Can you give me a list of different sub-genres of crime fiction?"). This factor may impact the outcome of the investigation in the sense that a crime novel set in ancient times may have more in common with a historical novel than a Tartan noir text set in contemporary Glasgow, let's say. 

Another point worth mentioning is that since I will compare crime novels written over the course of 120 years using lemmas, it may be that the genre has changed over time to such a degree that earlier books of the genre have more in common with, inter alia, adventure fiction than modern crime fiction. So, part of this exercise is to discover crime novels that do not conform to the genre and then in follow-up studies try to figure out why this is the case.

The model and preprocessing

The Doc2Vec model built using Gensim adopted for the investigation had these parameter settings: vector_size=300, window=5, min_count=2, workers=4, epochs=30. The input was all the lemmas of the texts with proper nouns converted to the tag propn and punctuation deleted from the input.

I started out by comparing all the texts (novels) in my corpus with all the other texts to find the average cosine similarity (CS) of the texts in corpus. Remember that we, in one of the earlier experiments, had tested various CS output thresholds, to get at the more potentially interesting similarities between two texts. The average, not surprisingly, is very low, 0.12, which shows, I think, that fiction texts are essentially quite different from each other, at least their lemma inventories. This pre-processing step, where I compare all the texts with each other, lets me extract the cosine similarities I will look into in more detail.

The texts

In this study, I could have compared all the texts tagged as crime in the corpus, but there are more than 500, so I chose 24 ones spread over the the span of the corpus (1900-2019), i.e. two from each decade. NB! The a priori genre labels attached to these texts were set by me, in most cases based on information found for the texts/novels on Wikipedia.

Apart from 1900-1919, I was able to pick one female and one male author from each decade. Here are the texts I chose, manually, but randomly from all the texts labelled 'crime' in the corpus:

The steps in the investigation

An important point to make at this point is that the corpus, more often than not, contains several books by the same author. This may boost the mean CS score, since we can expect two books by the same author to achieve a high(er) similarity score than when the same book is compared with another author (writing perhaps also within another genre).

The basis for the study is the tab-delimited CSV file where I have all the CS comparisons, i.e. where I have compared all the texts with each other. From that file I extracted the comparisons related to each of the 24 texts, excluding, of course, where the text was compared to itself, and created 1,719 files (= number of texts in the corpus), again in a CSV format, with the name of the author of the text, the text code in brackets, its genre and the CS. One row in the file named 'ArthurConanDoylepg2852_crime.txt' for instance looks like this:

• A.J. Quinnell (PhiNic2)<tab>crime<tab>0.1208. ( A.J. Quinnell is the pen name of Philip Nicholson); another row will be
• Alan Sillitoe (AlaSil1)<tab>general<tab>0.0569

The next step is to count and find the mean CS score for each of the genres of the Arthur Conan Doyle text, i.e., to find out if, according to our hypothesis, the label crime will get the highest mean score. For this particular Conan Doyle text, this is not the case as 'adventure' gets a mean score of 0.1862, 'historical' has a mean score of 0.1347 and 'crime' 0.1336. So, compared to the texts of the CBF, this particular Conan Doyle texts has more in common with the adventure novels of the corpus when we compare the lemma inventories of the texts using cosine similarity. If we, on the other hand, look at the maximum CS this text got when compared to the other texts in the corpus, another crime novel gets the highest score, 0.4176 (the highest score on an adventure text is 0.3406). The max. score comes from the comparison with an Arthur Morrison novel titled "The Red Triangle", published in 1903. Since I set a minimum score threshold of 0.4 to reduce output, this novel was the only one that reached the threshold.

Next, I present the result of the comparisons for all the 24 chosen texts. But before I do that, here is a table showing the number of texts in each genre. The higher the number, the more solid the basis for the comparison can be claimed to be. There are also important diachronic factors impacting the CS values over the century, e.g. the fact that the corpus has no texts categorised as adventure fiction after 1969. The genre category 'war' is the smallest in number of texts and is unevenly distributed over the span of the corpus with no instances in 1980-1989, 2000-2009 and 2010-2019.

The results

The below table summarises the findings, highlighting some of the (perhaps) unexpected outcomes and results of the comparisons.

The first thing to notice is that 'crime' is not the genre with the highest mean CS score for 7 of the first 8 novels, even though the max. score, i.e. the score obtained when comparing with a particular novel, is always 'crime' for these as well. We hinted at this when we wrote above that the crime genre may have changed over the 120 years the corpus spans, at least when we compare the lemma inventories of the texts. 

For two of the texts, the genre label 'general' have the max. score. Below we will reveal which texts these are, to see if they, e.g., are mis-categorised somehow. A final point is that 'humour' seems to figure prominently in quite a few of the books, at least those between 1930 and 1990.

The following table lists, in the second column, which author the the primary text resembles the most in terms of CS score based on lemmas. Columns three, four and five shows respectively the genre of that second texts, the CS score of the comparison and, finally, if an author writing within another genre reached our threshold of >= 0.4. 'None' means that no other text in the corpus reached this threshold, when compared with the primary text of column one.

The first, and obvious, observation we can make is that the highest similarity scores  come from comparing two text by the same author. There are only three exceptions to this "general" rule. These are the texts by Arthur Conan Doyle, Dorothy L. Sayers and Minette Walters. The Minette Walters text is doubly interesting in that the most similar book is the general genre novel "The House of Sleep" by Jonathan Coe. Secondly, the table goes some way towards establishing 'crime' as a coherent genre in that many of the comparisons only reach a CS score of >= 0.4 when the primary text is compared to other crime novels.

Another interesting finding, not shown in the table, is that the final three books have more than 30 texts that reach the CS score threshold I set at >= 0.4. As we can see from the table, this is not the trend since quite a few of the original texts have no other text reaching the threshold. 

In one aspect, the resulting table brings us back to our experiment 1 and 2 where we showed how texts written by the same author, even if writing under a pseudonym, got CS scores >= 0.6.

In my next post, I will get under the hood, so to speak, of some of the cosine similarities we have seen to try and tease out what it is in the texts that triggers the scores.