On his death in 1727, pioneering physicist Isaac Newton left behind a trove of manuscripts that he had shared with almost no one during his lifetime. The long-unpublished papers—containing some 10m words, or the equivalent of roughly a hundred novels—have tantalised scholars ever since. Newton famously left little published evidence of how he made his scientific discoveries. Could these private writings hold the key to understanding his genius?

Recovering the gold within his papers is no easy matter. The material is difficult on many levels. Forbiddingly technical, and unrelentingly heterogeneous, these are the writings of an introverted scholar working across six decades who was loath to throw away even the smallest scrap. One measure of the difficulty of this material is the fact that no comprehensive edition of Newton’s writings has ever been published.

To make matters worse, Newton left no will and no instructions for dealing with the papers. It was a strange omission that becomes clearer once it is understood just how inflammatory their contents were. The papers contained damning evidence of Newton’s heretical disbelief in the notion of the Trinity of God the Father, Son and the Holy Ghost—which he believed to be a mere scriptural corruption. Newton’s fascination with alchemy was also evident from these papers, as was his unseemly obsession with abstruse matters of church history and doctrine.

Read more about science:

The replication crisis

Neutrino astronomy: Seeing without light

How to predict the future

Immediately following Newton’s death, his relatives brought a Fellow of the Royal Society in to make a quick assessment of their worth (he had just three days with the papers). Unsurprisingly, he determined that, with a scant few exceptions, the papers were “Not fit to be printed.” The existence of these papers threatened Newton’s public image as a scientist-saint. His heirs soon decided that the best policy was one of genteel hush. The papers were packed up and kept in the family estate where they remained inaccessible to all but a few for more than a century.

Yet knowledge of their existence persisted and as the years passed, scholars never stopped hoping to unlock their secrets. In 1841, the logician, mathematician and pioneering historian of science Augustus De Morgan considered the matter, reflecting on the need for a handwriting specialist able to decipher Newton’s (in fact fairly neat) writing. “When will it happen,” he wondered, “that a paleographist is also a mathematician, with enough energy and leisure both to work the ore and the metal?” De Morgan’s metaphor captures the urge to mine the brilliant scientific work and separate it from the unsavoury material—and the serious work that would be required to do so. Only in the 1870s was there finally a real chance, when the Earl of Portsmouth decided to donate the scientific material to the University of Cambridge.

The papers were in such a jumble that the whole lot—scientific and non-scientific—was sent to Cambridge where a committee was set up to catalogue and sort them. Headed by the co-discoverer of the planet Neptune, John Couch Adams, and Newton’s successor as Lucasian Professor of Mathematics, George Gabriel Stokes, that committee failed spectacularly to engage with the key feature of Newton’s great archive: its alchemical and theological dimension.

This was not for lack of time, or through simple ignorance. The committee took 16 years to complete its task, during which time both Stokes and Adams in their own ways grappled with the papers. Stokes was famously prolix, disorganised and dilatory. The result was that he very nearly lost the precious Newton manuscripts amid the chaos of his study, which he’d crammed with as many tables as he could “with narrow passages between, through which to squeeze if you could.” The tables were piled with papers more than a foot high (Stokes’s wife sometimes removed “clothes-baskets full of unnecessary material” when he was out of the house). Little wonder that Stokes kept the Newton material for “so long that there was some anxiety as to whether they had been overlooked.”

Adams, meanwhile, had his own weaknesses as an editor. Famously averse to writing (he did his most important calculations in his head), Adams was a perfectionist. The Newton material with which he grappled was scientific, not religious or alchemical, and Adams gave it the full measure of his attention. The process took years, as he managed to unravel the “enigmas” that the great man had scattered across “stray papers, without any clue to the source from which they derived,” as Adams’s friend recounted. For any other man, such a feat would have been impossible, but Adams’s “mind bore naturally a great resemblance to Newton’s in many marked respects.”

Not in all respects, however. He and Stokes gained some understanding of Newton’s scientific genius from the papers (more a product of hard work, they concluded, than sudden inspiration), but they steadfastly ignored the millions of words of alchemical and religious writings he’d left behind as “of very little interest in themselves.” It was only in the 1950s that the bulk of Newton’s writings, both scientific and non-scientific, finally became available to professional historians with both the skills and the determination to take their full measure. Even then, the task was daunting.

Scholarly editing is lonely, painstaking and often tedious work, rarely acknowledged and generally ill-paid. Furthermore, it is existentially challenging. In order to interpret unfinished, ambiguous and often contradictory writings, editors must get as close as possible to their subjects. They must study what their subjects studied and learn how they thought, but then they must stop themselves from overreaching acts of imaginative transference, from presuming to think on behalf of their subject. Scholarly editing requires very clever people to imaginatively subordinate themselves to even cleverer people.

With Newton, the paradigmatic genius, the problem is especially acute. In 1924, one prospective editor, RA Sampson, described the challenge posed by Newton’s scientific manuscripts thus: “Before anyone undertakes it he should see clearly that he is making a voyage into the past, the greater part of it completely dead and obsolete. An editor must allow himself to be absorbed into this defunct world for many years. Probably he would never quite find his way out of it.”

Even 30 years later, the editor of Newton’s correspondence, EN da C Andrade, was dyspeptic about the likelihood of ever succeeding in the task. “A large, possibly very large, proportion of this manuscript matter is, then of little value,” he said, “and would certainly be rejected altogether by wise editors; but it is hard to suppose that there are not treasures among it.”

By the 1970s, as editors continued their work, it was clear that Newton’s non-scientific work made up the bulk of his writings, and was impossible to dismiss out of hand. Understanding Newton wasn’t simply a matter of sorting and discarding the non-scientific material. Instead, it became imperative to understand the relation between the scientific and non-scientific aspects of his thought. How to account for a Newton of so many parts? What, if anything, linked the disparate material together? For some, the notion that the science, theology and alchemy might be interlinked was distressing. For others, the absence of a unity of thought was even more so.

Debates on who Newton was—and what his papers might reveal about him—have swung backwards and forwards over the course of the nearly 300 years since his death. After a period in which scholars sought to find connections between the scientific and non-scientific material, scholars today are more willing to allow that Newton himself maintained boundaries between his ideas and methods on different subjects. Despite these swings, the history of the papers is also remarkably repetitive. The “surprise” of Newton’s dark side is a recurrent one, which has been discovered and re-discovered by scholars and editors over the generations. The image of this secret Newton, cultivating mystical and heretical beliefs behind the curtain of his public self turns out to be just as enduring as that of the scientist-saint. Sorting out Newton is even harder than it looks.

What would Newton make of all of this? Doubtless he would be horrified that his most private, and messy, thoughts have now been made public by scholars. Nearly all of all of his religious and alchemical writings have now been digitised and are freely available online. But Newton himself saved the papers, presumably in the hope that one day they would be read by those who were prepared to accept the hard truths they contained. While that day has not yet arrived, today the private Newton has become irrevocably public.

The Newton Papers: The Strange and True Odyssey of Isaac Newton’s Manuscripts is published by Oxford University Press and is available for £19.99