Introduction
In the earlier experiments I have reported on, I have calculated CS scores at the level of whole texts, e.g. between books of the same genre. Now I move on to look more closely at what it is in the texts we compare that leads to these CS scores at text level.
In the post CBF and cosine similarity: The Crime Genre, for instance, we saw how books of the same author had markedly higher CS score than when we compared books written by two different authors, with some exception of course. More interestingly, at least for a linguist, is what it is in the texts that leads to these high(er) scores? We shall attempt to answer this by comparing text pairs at the level of the chunk, which, as we have discussed earlier, consists of either 100 or 1,00 words or sentences.
Method
The method I will employ is the same when we now compare chunks as when we compared whole texts, and I will stick with our 100-lemma chunks.
Preprocessing steps
In the post CBF and cosine similarity: Experiment no. 3, I altered all lemmas tagged as proper nouns (PROPN) to that tag (propn) to avoid "creating" differences between texts due to the use of different place and personal names. In this study, I have decided, wisely or not, to take this distinction one step further and make a distinction between different kinds of proper nouns tagged as PROPN. I will add the first part of the semantic tag to the propn part of speech tag. This means that when we have a proper name in the text, POS-tagged as propn and the accompanying semantic tag is Z1m or Z1f, the lemma, e.g. John, will be replaced by propn-Z1m in the output. In the corpus, words tagged as PROPN will often have the semantic tag Z1mf or Z99 (= 'Unmatched', i.e. the semantic tagger does not know what this is). So, what do we do about these? My decision for Z1mf is to remove the 'mf' part. As for Z99, after looking at the output of the tagger, it seems that the Z99 tag mostly represents a proper name, so when the POS tag is PROPN, I will change the semantic tag Z99 to Z1 for this study. Here is an example of how this look before and after post-processing the output of the spaCy tagging pipeline.
The original text snippet:
... Since Lily’s death and Teiji’s disappearance, I have become nervous about many things. I pulled open the wardrobe door and crept beneath...
The output from the PyMUSAS semantic tagger:
The output after post-processing and input to the Doc2Vec model and CS score calculation:
... since propn-Z1f 's death and propn-Z1 's disappearance I have become nervous about many thing I pull open the wardrobe door and creep beneath the ...
Similar outputs can be produced containing words (tokens), part of speech or semantic tags, depending on what we may want the model trained on. And, as I wrote in an earlier post, we can choose to employ a stopword/-lemma list (disregarded here) and whether to include or exclude (as here) punctuation in the output.
The experiment
To be sure to get a relatively high CS score when comparing two 100-lemma chunks from two different texts (novels), I chose two texts from the same author, i.e. Michael Dibdin, labelled MicDib1 and MicDib2. I then compared all the chunks in MicDib1 with all the ones in MicDib2 and outputted the chunks with a CS score >= 0.7. Remember that a CS score of 0.7 is considered high when comparing fiction texts.
Table 1 shows the two chunks with a score of 0.71. The table has both the original text snippets, with a bit more context (2nd row), and the actual chunks that was output by the cosine similarity comparison script (3rd row). I have highlighted some words/lemmas, which I will comment on below.
| MicDib1 | MicDib2 |
|---|---|
He turned to the third occupant of the compartment, sitting by the window, a distinguished-looking man of about fifty with a
pale face whose most striking feature was a nose as sharply triangular as the jib of a sailing boat. There was a faintly exotic air about him, as though he were Greek or even Levantine. His expression was cynical, suave and aloof, and a distant smile flickered on his lips. But it was his eyes that compelled attention. They were grey with glints of blue, and a slightly sinister stillness which made the Veronese shiver. A cold fish, this one, he thought. ‘What about you, signore?’ he demanded. ‘Don’t you agree that it’s a disgrace, a national disgrace?’ | Grimaldi was just congratulating himself on his success, when he was called back from the antechamber where he had been sent to kick his heels and introduced to a newcomer, a man he had never seen before. Slightly taller even than Lamboglia, he had fine, slightly wavy hair and a face stretched as tautly over its bones as a drum. His angular nose and square, protuberant chin might have looked strong, but the mouth was weak and indecisive, as were the opaque grey eyes. Or so Grimaldi thought, until they turned towards him. It only lasted a moment, but he felt as though he had never been looked at before. ‘This is Dottor Aurelio Zen, a specialist investigator dispatched by the Italian authorities in response to an urgent request conveyed by
the apostolic nuncio,’ Sánchez-Valdés announced. |
| pale face whose most striking feature be a nose as sharply triangular as the jib of a sailing boat there be a faintly exotic air about he as though he be greek or even propn-Z1 his expression be cynical suave and aloof and a distant smile flicker on his lip but it be his eye that compel attention they be grey with glint of blue and a slightly sinister stillness which make the veronese shiver a cold fish this one he think what about you signore he demand do not you agree that it be a disgrace a national disgrace | he have never see before slightly tall even than propn-Z1 he have fine slightly wavy hair and a face stretch as tautly over its bone as a drum his angular nose and square protuberant chin might have look strong but the mouth be weak and indecisive as be the opaque grey eye or so propn-Z1 think until they turn towards he it only last a moment but he feel as though he have never be look at before this be propn-Z1 propn-Z1 propn-Z1 a specialist investigator dispatch by the italian authority in response to an urgent request convey by the |
An ocular scan of the two text snippets and their lemmatised chunks reveal that they have several words / lemmas in common. These common words no doubt contribute to the high CS score. Figure 1 is a wordcloud of the two chunks combined with stop lemmas excluded.
The lemmas that stand out in the cloud is another way of highlighting the lemmas that occur repeatedly. Finally, let us list the top four topics in these chunks. This time I did not exclude the stop lemmas:
- be, he, you, that, of, about, disgrace, and, his, as
- be, have, he, propn_z1, as, and, the, drum, square, response
- have, before, propn_z1, he, the, by, as, never, but, look
- as, his, face, slightly, grey, or, nose, though, he, the
These lists show how high frequency (function) words contribute to the CS score. Lemmas such as and, as, be, but, he, have, his, or and the are part of nearly all of the top four topics.
More help from AI
As part of this experiment I was curious to find out if AI could help me point to reasons for the high score apart from, or in addition to, what we can see by just having the two chunks side by side. So, what I did was to send in two prompts, 1. and 2. below, to both Copilot Chat and the University of Oslo's interface to GPT-5. Attached to the second prompt was the two chunks.
- I have built a Doc2Vec model consisting of fiction texts (British novels). The model is trained by feeding it text chunks of 100 words, i.e., each text has been cut up into smaller chunks of 100 words which are inputted as data to the model. I am currently comparing 100 words chunks from different texts with a cosine similarity of >= 0.7. Can you, without having access to the model itself, explain or guess what it is in these chunks that gives rise to the high cosine similarity score?
- Next are two chunks, separated by <p></p>. Can you analyse these for me along the lines you have stipulated?
The answer to the first prompt/question was, of course, 'yes' and a rather long (and a bit unnecessary) explanation of how a Doc2Vec model works. The response to the second prompt/question is more interesting and we will look at it in some detail.
AI's analysis of the two chunks
GPT-5 summarised the similarities in this manner:
"Bottom line
The high cosine similarity here is very plausibly driven by the overlapping physiognomic description vocabulary, the shared eye/gaze motif and associated affect, Italian cultural markers, the investigator/authority domain, similar narrative mode with interior thought, and the propn-Z1 placeholder artifact. These are exactly the co-occurrence patterns Doc2Vec exploits to place chunks close together."
Copilot Chat started its analysis by saying: "Both chunks describe **male characters** in a detailed, literary fashion."
Both summaries seem pertinent and to the point and, one might suspect, draws heavily on the underlying language model, which is the same in both cases, and its "knowledge" of fiction texts.
I attach the full responses from the two chat bots here: Copilot Chat / UiO's API interface to GPT-5.
Despite the several similarities that lead to the high similarity score, Copilot also points out the differences between the two chunks.
Lemmas unique to chunk 1:
- triangular, jib, sailing, exotic, greek, cynical, suave, aloof, sinister, shiver, cold, disgrace
These lean toward:
- Metaphorical and emotional language
- Dialogue and judgmental tone
Lemmas unique to chunk 2:
- wavy, tautly, drum, protuberant, indecisive, opaque, investigator, authority, dispatch
These lean toward:
- Precise physical description
- Professional or institutional context
These differences do not count enough towards lowering the CS score it seems.
One final point worth mentioning is that my decision to change proper names to propn-Z1(m/f) may have exaggerated the similarity between the two chunks, and GPT-5 suggests that I "count 'propn-Z1' frequency across the corpus. If it’s uncommon or unevenly distributed, its repetition will inflate similarity; if it’s a global placeholder used widely, the shared effect will be smaller but still present." I think the latter.