Stylometry

Sytometry sty·lom·e·try /stīˈlämətrē/
the statistical analysis of variations in literary style between one writer or genre and another.

FAQ

• What do you use to perform Stylometric analysis? Stylo is a package developed and maintained by Computational Stylistics Group for R Studio.
• Can I use Stylo with Python? PyStyl and faststylometry are two Python libraries inspired by Stylo.
• Where are the peer reviewed studies using Stylo? The Publications link on the Computational Stylistics Group website contains links to papers using Stylometry, Note: there are peer reviewed links on that page, but not all of the links are to peer reviewed papers.
• Does it utilize machine-learning? Several machine-learning methods are available for supervised runs of Stylo with the classify() function. Only unsupervised runs are utilized in these examples.
• What's the difference between supervised and unsupervised runs? Supervized runs used labeled data. Meaning you know the author beforehand and you're training their known works to identify if that person has written other works not attributed to them. Unsupervised runs compare the works provided without any influence from a specific writer. This allows the works to be compared to each other independently.
• Can it be used by AI/LLMs? Yes. It was used in one of the linked publications.
• Did Stylometry really help convict the Unabomber? Yes but they did it by hand.
• Why don't people do it by hand? Using a computer for the analysis eliminates virtually all human error (with the exception of bad/malformed data), bias, and is much faster than human analysis.
• Does the file name influence the results? No. Though, to control for a perceived bias, in the first example the file names have been obfuscated.
• Do you need to know the language being analyzed? You only need to be able to identify what language is in the file, so you can choose it appropriately in Stylo.
• Do you need to have writings from throughout the person's life to accurately identify them? No; the individual texts are compared to one another in unsupervised runs.
• What is the recommended amount of most frequent words (MFW)? It's recommended that at least 100-200 MFWs are used to get a strong authorial signal. However, there are instances where authorship can be determined with fewer words, and that can be observed by outputting the results in increments. Increasing engrams can also help identify someone through repeated phrases. This is demonstrated below.
• Where did you get the source data? I used two primary sources. For the New Testament I used the Society of Biblical Literature Greek New Testament (SBLGNT). The data is available in XML and can easily be split by chapter. Other Greek texts are available at Perseus at Tufts University. The text of Evangelion is from Github by Mark Bilby.
• How accurate is Stylometry? It has been shown to identify Greek genres with >97% accuracy. A new methodology has been demonstrated to have an accuracy of around 84.2%-89.6% for texts containing 50 words. The paper shows accuracy rates as high as 92% with 200-word passages. Eder (from Computational Stylistics) has found that 2000 words may be sufficient, but he has also said some authors can be determined with a 100 word sample. The paper he referenced can be accessed here. Take note: The samples he ran are in English, which is considered weakly inflective compared to the Greek and Latin demonstrated on this page.
• Does Stylometry prove Jesus didn't exist? Nobody is making that claim. However, it can be inferred indirectly. As it says in the definition above, it can identify authorship, including if someone was not the author. By identifying authorship and knowing when that person lived it logically follows that it was written during their lifetime, which in turn can also identify it wasn't written during someone else's lifetime if that person wasn't alive during the identified authors lifetime. It also assists in pinpointing interpolated texts and resolving the chronological debate of Evangelion predating Luke. Which is additional evidence for the 2ND century authorship of the New Testament.
• Are you a data scientist? I'm not a data scientist, but if the science is sound it is repeatable. I am a senior software engineer by trade but I have also had previous work peer reviewed by a data scientist. A process that took 8 months to defend and was determined to be error free on a dataset of several million records that was updated quarterly.
• Why not use Bayesian statistics? A design feature of Bayesian statistics is the use of non-zero probabilities in order for future updates; if something is impossible or completely rules out the likelihood or expected value, then it can't be used. This feature can be useful in some cases, but informative non-zero priors are subjective and can introduce a bias because something must be possible/probabile. Example: Bayesian statistics would require the possibility of being able to swallow my own head whole to return a non-zero result. Bayesian statics should also disclose their hypothesis/prior, evidence/likelihood, expected probability, and how those values were determined.
• Didn't Dustin White debunk the how Christ Before Jesus used Stylo? He tried to, but there were many flaws in his attempt.

Stylometry Examples

Influence of time

This shows the passage of time influencing writing style. There is an indisputable separation between all of the authors. Suetonius lived and wrote during Tacitus' lifetime (early 2ND century), and they're closer to Lucan (1ST century) than Tertullian (2ND-3RD).

This also shows us the passage in Annales 15.44 wasn't written by Tacitus.

 

Here is the file key.

Control Test

An example of a control test using a shuffled copy of a file. The shuffled text is on the left side and the original text is on the right. The preamble is not verbose enough to influence the results.

The difference in character counts is due to the shuffler using whitespace to split words and then joining by a single space. It doesn't influence the results, as you can see the distance is 0. They are grouped together at the bottom of the chart.

Testimonium Flavianum

Here is the Testimonium Flavianum compared to the works of Josephus and Eusebius. Antiquities of the Jews Book 18 used in the analysis contains the entirety of the Testimonium Flavianum. 3 engram/word phrases show an even closer connection to Eusebius. This demonstrates that even when the full text of a quote remains in the original, it's not enough to influence the result. The only exception would be if the quote makes up the majority of the text.

There is criticism that text can be faked if someone uses multiple word engrams to impersonate someone else. That could be true if someone was writing later and claiming to be Eusebius. In this case it turns out to be a smoking gun; Josephus never uses those phrases, but Eusebius clearly does. Obviously, it would be impossible for Josephus to copy phrases from someone in the future.

Eusebius writes in Praeparatio Evangelica Book XII (12) Section XXXI (31) that he believes in lying for the benefit of those who "require" it: "That it will be necessary sometimes to use falsehood as a remedy for the benefit of those who require such a mode of treatment"

The image below is a 3 engram dendrogram at only 100 MFWs.
As you can see, the results are similar to the 3 engram run at 2000 words.
The sample sizes are still more than what's recommended.
Total words: 413971
Average words: 13353.90322580645
Total unique words: 56806
Average unique words: 1832.4516129032259

* Note: Unlike the run above, this run does not include the TF in book 18 or the "so called Christ" in book 20.

Gospel of Luke

This is a consensus tree performed on Luke. The consensus tree requires at least three incremental passes on the data. For the consensus tree below I used four passes incremented by 500 words each time, hence the 500-2000 MFW label. The "consensus strength" value has been increased from the default setting of 0.5 to 0.9, a strength below 0.5 will return an error. This shows us that 1-3 and 23-24 are not written by the same author of the majority of the text. This is apparent on a dendrogram with as little as 100 MFWs.

Luke compared to Acts

The scholarly consensus is that Acts and Luke were written by the same person. That's partially correct; only the beginning and end of Luke were written by the author of Acts. The consensus strength of the tree was set to 0.99 in this case.

Who wrote most of Luke?

Many scholars believed that Evangelion was written after Luke and Marcion removed details from it. However, the data tells us Marcion wrote Evangelion first.

Luke was utilized by the author of Acts who padded it and added extra details to some of the verses.

If Luke and Acts were written by the same author and Evangelion stole from them, the bulk of Luke wouldn't have the clear separation from Acts and Evangelion would match with Acts.

We also see a distinction from the first 12 chapters of Acts and the last 11. Which might be from a second author or due to the shift in narration as it changes from third person to first person.

This highlights Marcion priority and Marcion is from the 2ND century.

Marcion's canon was assembled in the 2ND century and contained the Gospel of Marcion (Evangelion) and Apostolikon which consisted of ten letters of Paul. While the Modern New Testament canon wasn't established until the 4TH century.

"Genuine" Pauline Epistles

The Most Frequent Words (MFWs) parameter was set to 200, as the computational analysis in Stylo relies on word usage frequency. This parameter selection is informed by the observation that the majority of the files (excluding Philemon) average fewer than 200 unique words across their chapters. Targeting this average effectively eliminates rare terms that appear only once or twice with single-use vocabulary accounting for approximately 10% of total unique words, while preserving high-frequency vocabulary. This configuration yields satisfactory analytical outcomes and doubles the default baseline setting of 100 MFWs.

Here is a tab separated value (.tsv) file with all of the word counts per chapter and book. It is broken down by 20 words per row. In all of the examples 200 words (10 rows) retrieves all of the words used multiple times and words that are only used once, and in some cases all of the words in the chapter itself.

A chart for Philemon is not included since it is only one chapter.

Epistle Data

All Epistle Chapters Compared

The dendrogram encompassing all chapters of the Epistles reveals three main clusterings, with two exhibiting a higher degree of proximity to each other than to the third. Additionally, two to three sub-clusters are identifiable within these primary groups. Cross-collaboration is evident in Romans and Galatians between the top and middle clusters, whereas the initial and concluding sections of Romans appear to have been expanded by the most distant group. The top cluster accounts for the majority of 1 Corinthians, while the middle cluster is predominantly associated with 2 Corinthians. The bottom cluster of writers is exclusively responsible for, or provided the mojority via expansion to 1 Thessalonians and Philemon, as well as the larger portion of Philippians. This is consistent with Dr. BeDuhn's reconstruction containing a shorter version of Philippians that was subsequently expanded by the bottom group. It is demonstrated that they were responsible for the majority of the contents of chapters 1, 2, and addition of a 4TH chapter. The dendrogram reveals there is enough of the original text in chapter 3 that the authorship is placed into the first group. This is consistent with the pattern observed in Romans and the opening chapter of 1 Corinthians. This finding, combined with the substantial distance separating it from the other clusters, indicates that the bottom group introduced its additions and revisions subsequent to the work of the former two groups.

An examination of the consolidated dendrogram reveals that the majority of chapters maintain their structural proximity relative to the independent analyses, though certain deviations occur when the full corpus is integrated. These shifts signify a deeper lexical affinity with the later additions than the initial metrics suggest when evaluating individual epistles in isolation. Such patterns not only distinguish the presence of various contributors but also uncover a deliberate effort to harmonize these later insertions with the stylistic precursors. While these collaborative layers may remain obscured at the specific chapter level, the full comparative synthesis renders these editorial nuances distinct.

Stylo citation