CBF and cosine similarity: Experiment no. 2

Experiment / Research question

Can cosine similarity reveal interesting similarities between authors and genres?

Tools / Preprocessing

The tools were the same as in Experiment 1 and the models were trained with these parameters: vector_size=100, window=5, min_count=5, workers=4, epochs=30. 

I trained two models, one with lemmas as input and one with sematic tags, both consisting of all the texts in the corpus and 100 units (lemmas, sem. tags) per line of input. Then I compared every text with all the other texts in the corpus, outputting all text pairs with a similarity threshold above a certain limit, initially 0.8.

The results: Lemmas

I started with the model trained on lemmas and with a cosine similarity threshold of >= 0.8. Texts pairs written by Paul Harding and Michael Clynes reached the threshold and the same did texts by Douglas Reeman and Alexander Kent. This, I think, shows the potential of using cosine similarity to reveal shared sameness at some level between fiction texts, since we know that the writer Paul Charles Dominic Doherty uses the pen-names Paul Harding and Michael Clynes, among others, and the same goes for Douglas Reeman who also writes under the pseudonym Alexander Kent.

One pair of texts reached a similarity of 0.98, but these turned out to be duplicates.

That cosine similarity can be used to spot significant similarities between texts by the same author shows, I think, the potential of using this method in the field of corpus stylistics as a stepping stone on to more fine-grained scrutiny of pairs with a high similarity score.  

When I lowered the similarity threshold to 0.75, pairs of texts by Wallace Breem and Alfred Duggan reached this threshold, and so did texts by Alfred Duggan and Mary Renault. Interestingly, all these texts have been given the genre label 'historical', showing that cosine similarity also can be used to group texts according to genre.

Lowering the threshold to 0.7 resulted in 64 pairs of texts, pairs that it will be worth while to follow up, to see if they have some common factors, e.g. genre, gender and/or year of publication.

The results: Semantic tags

Using the model trained on sematic tags gives a slightly different result. Apart from the duplicates, no text pair reached the threshold of 0.8 or even 0.75. Lowering the threshold to 0.7 gives us these pairs (genre label in brackets):

• Paul Harding (crime) and Michael Clynes (crime): 0.728

• Ellis Peters (crime) and Edith Pargeter (general): 0.728

• C. Northcote Parkinson (historical) and Patrick O'Brian (historical): 0.726

• R. F. Tapsell (historical ) and Simon Scarrow (historical): 0.716

• Nicholas Monsarrat (war) and C. S. Forester (war): 0.701

We recognise the first pair, as this got a high similarity score when we used the lemmas as input as well. The second pair is quite interesting, as it did not pop up when using lemmas as testbed, especially since we know that Edith Pargeter uses the pen name Ellis Peters when she writes crime fiction. She is perhaps best known for her historical fiction, so maybe the genre label 'general' is wrong. Be that as it may, the similarity score seems to find that the two texts share many semantic features, which begs the question of how much variability there is between (some of the) novels written by Edith Pargeter. She has written more than 100.

The final three pairs are also interesting in that they show similarities between texts by different authors writing within the same genre. Can cosine similarity perhaps be used to group texts by genre? This is an intriguing question we have to come back to.

The results: parts of speech

As an afterthought, I decided to run the program using POS tags as input to the model too. Again, I set the threshold at >= 0.8 and, to my surprise, this gave me a list of 214 pairs, ranging from 0.93 to 0.80 in cosine similarity. The now, well-known pair Paul Harding and Michael Clynes came out on top, but the next 11 with similarities above 0.88 are perhaps more interesting and need further (qualitative) scrutiny.

What the results of the experiment clearly show is that the choice of input to the model, lemma, semantic tag or parts of speech has a great impact on the outcome.

(I asked Copilot if a cosine similarity > 0.6 would be considered large between two fiction texts, and the answer it gave was: For fiction texts, which can be quite diverse in style and content, a score above 0.6 typically indicates that the texts share significant thematic or stylistic elements.)