Introduction
To be able to perform the experiments reported on, I imported the following spaCy modules and packages:
- import spacy
- from spacy.matcher import Matcher
I also imported sys, os, re and json and loaded the spaCy model en_core_web.trf for best accuracy, set nlp.max_length = 2800000 (equals max length in characters of my texts) and matcher = Matcher(nlp.vocab).
The next step was to run the Python script with these settings on all my (now) 1,788 CBF texts. This took a few days on my CPU-only Fedora intel pc with 64GB memory.
The output from the script was one file for each text with the frequency counts of the following words (tokens): mother, father, sister, brother, son, daughter, grandmother, grandfather, granddaughter, grandson, aunt, uncle, niece, nephew, cousin, mother-in-law, father-in-law, sister-in-law, brother-in-law, daughter-in-law, son-in-law, stepmother, stepfather, stepsister, stepbrother, stepdaughter, stepson, great-grandmother, great-grandfather, half-brother, half-sister, he, she, mr, mrs, ms, miss, wife, husband, madame, missus, sir, mister, male, female.
Note that e.g. stepmother is sometimes written step-mother and grandfather grand-father, so this was taken into account. Likewise, for mr and mrs, which are often written mr. / mrs. and where the punctuation is treated as part of the lemma (abbreviation) by spaCy.
The final two variables, male and female, were used to hold the counts of first names attributed to male and female characters in the novels. NB! To match first names to gender, I used the package gender_guesser.detector. There probably exist more accurate packages, e.g. gendrize.io, but since my texts contain more than 28,000 names, I would have had to pay for access, which I did not want at this experimental phases.
To count the frequency of these words and names, I looped through the doc (doc=nlp(mytext)) and ents (for ent in doc.ents: if ent.label_ == "PERSON":). To be able to calculate the frequency per one million words, I also noted down the total number of words per text.
The frequency of use of 'he' and 'she' in novels written by male versus female writers
A straightforward and easy comparison to perform is to plot the use of 'he' and 'she' across the decades the corpus spans, i.e.1900-2019.
Figure 1. The use of 'he' and 'she' in texts written by male and female writers per million words
The main picture that emerges is that the frequency of use of 'he' by both male and female writers and the use of 'she' by female writers are pretty stable across the century. The use of 'she' by male writers, however, is clearly used less often, although we notice a rise from around 1980. In the same period, the use of 'she' by female writers drops slightly from around 1980. There is, moreover, an intriguing spike in the use of 'she' by male writers in the decade from 1920-1929.
Below are two tables containing the frequency of use of 'he' and 'she' in the Corpus of Historical American, Fiction, between 1900 and 1010. The rows above the bars are (1), frequency of 'he' and 'she' in the decade, (2) frequency per million words, and (3), total frequency of 'he' and 'she' per decade. Unfortunately, I could not count uses by male from female writers separately in COHA (chart view).
Table 1. The frequency of use of 'he'
Table 2. The frequency of use of 'she'
The COHA data cannot be compared directly with our data, but show a rise in the use 'she' from around 1970 and that the use of 'he' has declined steadily over the period.
Male writers use of 'she' in the 1920s
Let us briefly return to the use of 'she' by male authors in the period between 1920 and 1929. What causes such spikes in the use of very specific pronouns, or indeed, kinship terms more generally (see the next installment) in CBF? If we compare the decade 1920-1929 with 1930-1939, the first thing we notice is that the average number of the use of 'she' per text for the first decade is 957, while for 1930-39 it is only 562. Here we should note that the coverage is better in the earlier decade and that the corpus contains 122 texts published between 1920 and 1929 and only 105 for the later decade. The total number of words are 10,584,679 and 9,005,261 respectively.
Moreover, two texts from the 1920s stand out with more than 5,000 occurrences of 'she'. The text with the most frequent use of 'she' in the 1930s has 3,733 occurrences. Of the 105 texts published in the -30s, 15 (14%) contain more than 1,000 occurrences of 'she', while for texts published a decade earlier 48 texts, or nearly 40%, reach 1,000 or more instances of 'she'.
It seems likely that these differences can be attributed to genre. Of the 122 texts published in the 1920s, 15 are tagged as Romance and 32 as Crime. The comparable numbers of the ones from the 1930s are 2 and 49.
The text with the highest number of 'she' (5,605) is December Love by Robert Smythe Hichens, a romance novel published in 1922.