The Voynich manuscript (VMS, Beinecke MS 408) is written in a unique alphabet and nobody has been able to extract any meaning from it. Words in the manuscript are highly structured. Some people believe that this regularity could be related with lexical morphology. In 1991, the way in which the “circle” characters (o,a,y in the EVA transcription system) alternate with the other characters has been analysed by Jacques Guy, who came to the conclusion that the “circles” behave like vowels in a phonetic writing system. Whatever its cause, word structure is one of the phenomena that clearly prove that Voynichese is all but random.

The subject of this post is the comparison between two models of Voynichese words:

Jorge Stolfi’s crust-mantle-core context-free grammar;

a context-free grammar based on Emma May Smith’s syllable ranking model.

Since I have taken liberties with both models, anyone interested in the details should refer to the web pages of the two authors (linked below). Other word models have been proposed in the past, some of them are mentioned by Stolfi.

Code, grammars and input files used for this experiments can be found here: https://github.com/marcoponzi/vms_word_models

Jorge Stolfi’s model

Stolfi discusses his model in this June 2000 web page. He classifies most characters into three types (core, mantle and crust) that can only appear in a specific order. “Circle” characters can appear more freely in different parts of words. In this model, both core and mantle can occur symmetrically in two distinct sections of words, so the overall structure is actually made of five parts:

crust-prefix, mantle-prefix, core, mantle-suffix, crust-suffix

As Stolfi writes:

words are parsed into three major nested “layers” -crust, mantle, and core- each composed from a specific subset of the Voynichese alphabet:

core: t p k f cth cph ckh cfh

mantle: ch sh ee

crust: d l r s n x i m g [and ‘q’; ‘g’ and ‘x’ are not included in the the actual grammar] …

suppose we assign “densities” 1, 2, and 3 to the three main letters types above, and ignore the remaining letters. The paradigm then says that the density profile of a normal word is a single unimodal hill, without any internal minimum.

In other words,as we move away from any maximum-density letter in the word, in either direction, the density can only decrease (or remain constant). The possible density profiles (ignoring repeated digits) are

1 2 3

12 21 13 31 23 32

121 123 131 132 231 232 321

1231 1232 1321 2321

12321

As can be seen, the “core” is made of the so-called “gallows” (EVA:t, p, k,f) and “benched gallows” characters (EVA: cth,cph,ckh, cfh); “benches” (EVA: ch, sh) and “e-sequences” form the “mantle”; all other characters belong to the “crust”.

Reformulation of Jorge Stolfi’s model

Stolfi provides a formal definition of his model as a context-free grammar. He also links scripts for the processing of the grammar and word-lists to be used as input. I was unable to execute Stolfi’s scripts, but it was not difficult to semi-automatically translate his grammar into the format required by the Python parsing library Lark.

The input file also had to be minimally pre-processed, in order to restore EVA:ee that had been replaced with “bh” (Stolfi’s grammar expects the usual “ee”). Also, the total number of tokens in the input file is not perfectly identical to the number of tokens listed in the grammar file (35027 vs 35128). But the success rate produced by Lark is extremely close (actually slightly better) than than that listed in the grammar file (33803/35027=96.5% vs 33827/35128=96.3%).



Stolfi’s grammar file includes frequencies that are used as weights for the rules:

The text is contaminated by sampling, transcription, and possibly scribal errors, amounting to a few percent of the text tokens — — which is probably the rate of many rare but valid word patterns. Thus, a purely qualitative model would have to either exclude too many valid patterns, or allow too many bogus ones. By listing the rule frequencies, we can be more liberal in the grammar, and list many patterns that are only marginally attested in the data, while clearly marking them as such.

The experiments discussed here focus on the binary classification of words as valid or invalid: the grammar I used does not include frequencies, which moreover I wouldn’t know how to represent in Lark. My preliminary results showed that Stolfi’s original grammar is indeed too “liberal” accepting a large number of non-Voynichese words. So I removed low-frequency rules until I obtained results with an overall quality comparable to that of Emma May Smith’s model.