In the summer of 2012, a number of philosophers at British and American universities received a bulky, unmarked package in the post. It contained a 560-page book, written in English but with the Latin title Summa Metaphysica, by an amateur whose name they didn't recognise: David Birnbaum. It isn't unusual for philosophy departments to get mail from cranks, convinced they have solved the riddle of existence, but they usually send stapled print-outs, or handwritten letters; Summa Metaphysica stood out "for its size and its glossiness", says Tim Crane, a professor of philosophy at Cambridge. The book was professionally typeset. It even included endorsements from Claude Lévi-Strauss, the legendary French anthropologist, who described it as "remarkable and profound", and from the Princeton physicist John Wheeler, who once collaborated with Einstein. It would later transpire that 40,000 copies were in circulation, a print run any academic philosopher might kill for. The book claimed to have sliced through countless fundamental problems in philosophy, physics and theology, and there on the spine, where the publisher's name appears, was one deeply reassuring word: "Harvard".

Then the story grew stranger. In May this year, the US-based Chronicle of Higher Education reported that prominent scholars – scientists, philosophers and theologians – had been persuaded to attend an expenses-paid "international academic conference" at Bard College, a respected institution in upstate New York, devoted to Birnbaum's work. "We are especially pleased to announce that David Birnbaum will be present during discussion," the invitations glowingly explained. They hinted that his work might point the way toward a reconciliation of science and religion.

But the event itself, on Bard's leafy campus beside the Hudson river, proved disorienting. It was "definitely, absolutely the strangest conference I ever attended", the astrophysicist Marcelo Gleiser told the Chronicle. Tammy Nyden, an expert on Spinoza, the great rationalist of 17th-century philosophy, "felt hesitant about the invitation to begin with", the Chronicle reported, "but because it was taking place at a venerable institution like Bard, she decided to go". On the one hand, Birnbaum's work had attracted plenty of credible endorsements: a typical blurb for Summa Metaphysica, attributed to a mathematician at Warwick University named Hugo van den Berg, described it as "unparalleled and magisterial". On the other, nothing about Birnbaum's approach was conventional. Conference-goers were surprised to find him handing out Summa Metaphysica T-shirts; it subsequently emerged that he had provided thousands of dollars of his own money to fund the gathering. Nyden recalled feeling uneasy: "Here's someone with a lot of money," she thought, "and they're buying a lot of legitimacy."

The academics could be forgiven for never having heard of Summa Metaphysica's author. But, in fact, he was far from unknown: David Birnbaum is a prominent figure in the New York jewel trade, a private seller of high-carat diamonds and other rare gems, with a clientele that has included celebrities – Goldie Hawn, James Gandolfini – but consists mainly of the anonymous super-rich. For some time now, aided by his wealth, Birnbaum has been on an altogether different mission: to convince the world he has made an astonishing breakthrough in philosophy. It is a quest that has seen him accused of "academic identity theft", epic levels of arrogance, and the unauthorised use of Harvard University's trademarks. But it also raises fascinating questions. These days, only a tiny number of people understand enough theoretical physics, or advanced philosophy, to grasp what these disciplines tell us about reality at the deepest level. Is it still conceivable – as it was a century ago – that a gentleman amateur, with some financial resources, could make a real, revolutionary contribution to our understanding of the mysteries of the universe?

There is no shortage of people who would say no, at least in Birnbaum's case. His work, said a commenter on the Chronicle's website, "reads like L Ron Hubbard had drunken sex one night with Ayn Rand and produced this bastard thought-child". One scholar who became professionally involved with Birnbaum described the experience as "unsettling, unfortunate and, to my knowledge, unprecedented in academic circles". Another just called him "toxic".

But then again – as Birnbaum pointed out to me, more than once, during the weeks I spent trying to figure out exactly what he was up to – just suppose that a scrappy, philosophically unqualified Jewish guy from Queens really had cracked the cosmic code, embarrassing the ivory-tower elites: well, isn't this exactly the kind of defensive response you'd expect?

A detail from one of Jim Carter's circlon diagrams Photograph: James Carter



The science writer Margaret Wertheim has made a specialism of studying people she calls "outsider scientists": obsessive amateurs, usually with little or no university education, who assert that mainstream science has taken a wrong turn, and devote themselves to constructing elaborate alternative theories of reality. The star of her 2011 book on the subject, Physics On The Fringe, is a trailer-park owner from Washington state named Jim Carter, who rejects quantum physics, arguing that the universe is actually composed of minuscule doughnut-shaped particles called circlons. Whatever else may be said about this theory, Carter's painstaking, multicoloured circlon diagrams are gorgeous; Wertheim once curated an exhibition of them at the Santa Monica Museum of Art.

Wertheim is unashamedly sympathetic toward her cast of eccentrics, a fact that led some critics to misread her as arguing that their ideas ought to be taken seriously. What she really wants us to take seriously, though, is the motivation behind their efforts: their insistence that the deepest secrets of the universe, as she puts it, "ought to be understandable by an ordinary, thoughtful person, who's willing to do some contemplating". Science is supposed to explain the world to us, turning shimmering mysteries into intelligible truths. But, in practice, few of us will ever understand the cutting edge of a field such as physics, because it requires so much advanced mathematics; we must take it on trust. "What happens to a society when the official cosmology, the official picture of the world, is literally incomprehensible to 99.9% of people?" Wertheim wonders. "On some level, isn't that just a very unhealthy situation for a society to be in?"

Summa Metaphysica seems to have sprung from this same insistence that the world should be figurable-out. It is nothing less than an effort to answer the most brain-bending question of all: why does anything exist in the first place? William James, brother of Henry, called this question "the darkest in all philosophy"; the astrophysicist Bernard Lovell warned that thinking about it could "tear the individual's mind asunder". "Like all deep incomprehensibilities," writes Jim Holt in his recent book Why Does The World Exist?, "it lends an opening to jocularity." When Holt put the question to the American philosopher Arthur Danto, he shot back: "Who says there's not nothing?"

Birnbaum isn't joking, though. Summa Metaphysica is actually two books: a 270-page preliminary volume, then the 560-page main event. (He has also published at least 15 ancillary works and operates, by my count, at least 12 websites, including philosophy1000.com, womb1000.com and potential1000.com.) It is an exhausting read, partly thanks to its length – volume two alone has 90 appendices – but also because much of it is written in a kind of rapturous, mystical prose, liberally peppered with capitals. A typical sentence reads: "The cosmic trajectory is from the bottomless VOID to the limitless EXTRAORDINARY." Birnbaum's big idea is what he calls "the Quest for Potential theory", or Q4P, or occasionally Q4P∞. The sense that he is unveiling hidden, pan-historical connections sometimes gives his work the flavour of Dan Brown.

The Chronicle article ignited a small but fierce controversy in philosophical circles. Bard College stood accused of taking cash from a charlatan, to let him purchase academic credibility, which was symptomatic, some argued, of the dominance of money over intellectual integrity in American academia as a whole. Harvard University announced that it was considering action against Birnbaum's alleged "infringement of the Harvard trademark". But Birnbaum wasn't cowed: after the conference, he emailed approximately 2,000 philosophers around the world, inviting contributions to a series of books on his work. They would be edited, he explained, by a Bard College philosopher named Garry Hagberg. Hagberg issued a statement vehemently denying he'd be doing any such thing.

Given all the fuss, I thought Birnbaum might have gone to ground. In fact, he proved strikingly eager to see me.

'People in academic philosophy are a little hostile,' says Birnbaum. 'Who the hell is this guy, coming and telling us we got it wrong?' Photograph: Brian Finke for the Guardian



David Birnbaum runs his jewellery business, as well as his secrets-of-the-universe business, from a large open-plan office on the seventh floor of a building on East 48th Street, in the heart of midtown Manhattan. It has a high-security double-door entry system: as you step inside, the first door clicks shut behind you, and cameras scrutinise you for signs of intent to steal diamonds, before a second door buzzes open, into a plush corridor lined with frosted glass. Birnbaum, who is an energetic 63, with dark hair turning silver, was waiting for me, beaming. He was dressed impeccably, in a dark suit complete with pocket handkerchief. A few members of staff were working industriously on what seemed to be gem-related business; a large safe stood in one corner. We chatted about diamonds for a while, but Birnbaum was impatient to talk about Summa Metaphysica. "Down to business! Down to business!" he said, meaning the meaning of life.

I asked about the controversies he'd sparked. Birnbaum didn't deny using the word "Harvard" on his books, or helping pay for the Bard conference, but it soon became clear he regarded these as distractions, and to be expected, given the tectonic implications of his theory. "People in academic philosophy are a little hostile," he said, speaking rapidly. "They're hostile to me as an outsider. Who the hell is this guy, coming and telling us we got it wrong? Who does he think he is? Basically, he should go kill himself, and the sooner the better!" His publishing house, he explained, was called Harvard Matrix; he considered it absurd that anyone might confuse it with Harvard University Press. (More recent editions of the books have "Harvard Matrix" imprinted on the spines, and he now calls the imprint "New Paradigm Matrix".)

He swore the celebrity endorsements were all genuine. (It's hard to assess the veracity of the Wheeler or Lévi-Strauss quotations, since both are dead. Of Lévi-Strauss, Birnbaum said: "I had correspondence with him over the years.") "To my mind, this is beneath my level of discussion," he said. "It's beneath your dignity, also." His exasperation was understandable. He believed he had worked out what made the universe tick. How could questions about publishers' logos or cheques for a few thousand dollars be anything but annoying?

To grasp why a successful New York jeweller, with little philosophical or scientific expertise, might want to probe such questions, it is illuminating to consider Birnbaum's early life. He had been haunted by these grand mysteries, he told me, since the age of 11, when he attended an Orthodox Jewish school, or yeshiva, in Queens. It was the early 1960s and many of his classmates were the children of Holocaust survivors, or other Jewish émigrés from Nazi Europe: humanity's capacity for great evil loomed large in recent memory. Yet the yeshiva boys were urged daily to put their faith in a just and merciful God. The contradiction that weighed on the young Birnbaum was the ancient theological puzzle known as the "problem of evil": how could God be just and merciful, yet allow something like the Holocaust to happen? The secular side of the curriculum proved equally dissatisfying. If everything began with the Big Bang – a term coined just a few years previously, in the 1940s – then what caused the Big Bang? If evolution explained how living things changed, why did life start to begin with? Why was there anything?

"So, pretty soon, it becomes clear to me that I'm not going to get answers," Birnbaum said. "Everybody's smart. Everybody means well. But we never quite get there." Through college, and on to an MBA at Harvard Business School, the questions never stopped nagging. "There must be an answer," he remembered thinking, "but how is it possible that so many brilliant people, over thousands of years, have missed it?" That was when he began to suspect the answer might have remained hidden not because it was too complicated, but because it was too simple: "I decided it must be hiding in plain sight."

The answer, after years of fruitless reflection, dawned unexpectedly. Birnbaum was in Barbardos on holiday in 1982, sunbathing on a beach and turning matters over in his mind. "I'm good on the beach," he explained. "My brain is working a little better… And then" – he snapped his fingers – "it was clear to me." The answer was: potential.

This part takes a little explaining.

Birnbaum considers his specialism to be metaphysics, that hard-to-define corner of philosophy that deals with the most basic questions of what there is. It's the territory into which you cross when you reach the limits of what biology, chemistry or physics can tell you. Metaphysical explanations aren't supposed to be substitutes for scientific ones, though; they just claim to be even more fundamental. And what could be more fundamental than potential? What must have existed, before everything else, but the potential for all those things that later came into existence? If you believe in God, the potential for God must have been there first. And prior to the Big Bang, there must have been the potential for the Big Bang.

Rising from the Barbadian sand, Birnbaum saw the world in a new light: everything and everyone around him was an expression of cosmic potential, working itself out. Why? Because that's what potential does. Birnbaum calls this process "extraordinariation". It is explained in depth in the hundreds of pages of Summa Metaphysica, but the core idea is concise enough to fit on a T-shirt. The universe itself is potential, actualising itself.

You may be raising your eyebrows at this. But Birnbaum's perspective isn't without precedent. Since Aristotle, some thinkers have been drawn to the notion that the world must be heading somewhere – that there is some kind of force in the universe, pushing things forward. These teleological arguments are deeply unfashionable nowadays, but there's nothing inherently unscientific about them. In his controversial 2012 book Mind And Cosmos, the US philosopher Thomas Nagel argues that teleology might be the only way to account for the still unsolved mystery of why consciousness exists. Still, as Birnbaum explained his theory, I must have looked underwhelmed, because he leaned forward in his chair to emphasise his point. "It works!" he said. "It's powerful! And with all due respect to Harvard, Oxford, etcetera… it's more powerful than anything you got!"

He raced back from Barbados to New York, took up residence in a library and began work on the book. There followed years of attempts to get it noticed. As the decades passed, his jewellery income grew, and his wife gave birth to three children. (They have since divorced.) There were some bites of interest in his writing: the first volume of the book, which focuses on theology, has been used for teaching in some Jewish colleges. Still, it was tough going. Many of Margaret Wertheim's "outsider scientists" operated on shoestring budgets, and had he faced similar constraints, it's conceivable that Birnbaum might have considered giving up. But that wasn't the case. Thanks to the diamond business, he had money.

Outsider thinkers are fond of arguing that the doors of knowledge are jealously guarded by a "priesthood" or elite, intent on excluding non-members. "And they are right," says the astrophysicist Marcelo Gleiser, who attended the Bard conference on the basis of what he feels was a misleading invitation. He regularly receives screeds from amateurs, claiming to have reshaped physics. "We need an incredibly good filter, so we don't waste all our time evaluating work like his," he says of Birnbaum. The system of peer review, and strict controls on who gets to take part in debate, let scholarship move forward, so that academics don't keep reinventing the wheel. (As if to underline this, Jim Carter, of circlon fame, independently rediscovered a theory about the motion of atoms from the 1880s that had since been disproved.) As a critic of Wertheim's book put it, "You might as well study French literature and not learn how to read or speak French, [then] whine about the fact that your ideas are not taken seriously by the 'priesthood'."

Ludwig Wittgenstein was an engineering student when he began reflecting on philosophy Photograph: Hulton Getty



But could philosophy be different? Tim Crane, at Cambridge, argues that it could. In science, the days are long gone when Michael Faraday, who started out as a bookseller's apprentice, could teach himself enough to revolutionise a field. But "the questions that philosophy asks are questions that my 11-year-old nephew could understand, and I think that's significant," says Crane. If there is a God, who made God? But he adds: "Philosophy is a discipline. You've got to discipline your thought. It's not just making stuff up. And disciplining your thought is very hard to achieve."

Even so, amateurs have managed it. Ludwig Wittgenstein was an engineering student when he began reflecting on philosophy, and if we no longer categorise him as an outsider, that's only because his work proved so persuasive. Copernicus and Galileo were spurned by the mainstream, but we never hear about the countless outsiders whose ideas rightly sank into obscurity. Anyway, the problem with theories such as Birnbaum's is not that they're ridiculous, Crane argues; it's that they don't go deep enough. "Lots of philosophers have thought of potential as being something that really exists," he says. "But if it's going to explain anything, it has to be something real. And if it's something real, then it can't explain how reality itself came into existence." Philosophy's darkest question persists.

Bard College, which was founded in 1860, sits on the eastern bank of the Hudson river with views of the Catskill mountains, in the tiny town of Annandale-on-Hudson, a two-hour drive from Manhattan. It has a student body of about 2,500, but its reputation exceeds its size, thanks partly to its music conservatory and to Leon Botstein, a prominent conductor who has been the college's president since 1975. It said something deeply impressive, in other words, that Bard might hold a conference dedicated to Birnbaum's Summa Metaphysica. Yet that conference, which took place in April last year, may not have been quite what it appeared: in hindsight, it seems a testament more to Birnbaum's knack for self-promotion than to any acceptance of his ideas. Garry Hagberg, the Bard philosopher who chaired part of it, said he had been told the conference was about science and religion, not Birnbaum; he agreed to take part as a favour to a colleague. That colleague, a theologian named Bruce Chilton, had befriended Birnbaum some years earlier at a seminar in New York; he accepted a $7,000 donation from him for the Bard event, but said he never agreed it would focus on his work. (It is not uncommon for private individuals to help fund events at private US universities.) Hagberg and Chilton never even saw the impressive invitations that Birnbaum had designed at his own expense. "He has financial resources that dwarf the rest of us," Hagberg said, sounding a little resigned. During one conference discussion, when Birnbaum casually referred to the event as being dedicated to his ideas, Hagberg was "completely stunned", he said.

"I think he was not used to being surrounded by academics," said Peter Atkins, the Oxford University chemist and prominent atheist, who was also present. "So when, in our academic way, we looked for flaws in his position, he really did get rather annoyed. He's got this view that the world is striving towards perfection, or something. But the second law of thermodynamics says that everything is getting worse: not a striving toward perfection, but an unwinding into collapse." Atkins had been sent Birnbaum's books in advance, and considered them without value. So why did he go? "Well, one reason is that I quite like going to New York," he said. "Another is that I do like engaging in argument… it keeps Alzheimer's at bay." Moreover, he added, "We were very generously looked after."

Not long after the conference, a professional contact in Britain alerted Hagberg to the email Birnbaum had sent to 2,000 philosophers, inviting them to contribute to a new project. (Birnbaum later agreed to retract the email.) "I felt like I'd been hit by a truck when I wasn't looking," Hagberg said. "This is 2,000 people, in my field!" He made the academics sound like flightless birds, unprepared for Birnbaum's self-promotional firepower. "Here, ensconced in the cloisters of academe, I think we just presume goodwill," he said. "The idea of sending out thousands of emails like that without asking the person named? In academia, you'd be ostracised for life."

The probability of any of these considerations dimming Birnbaum's enthusiasm seems close to zero. As far as he is concerned, he has perceived something big and true – the biggest and truest thing of all – and the criticisms and controversies will always be sideshows. True, he concedes, he hasn't worked out every detail of his theory yet. ("I'm a macro-conceptual theorist!" he protested when I raised this point.) But he's confident about it. Choose any outstanding scientific mystery – consciousness, the origins of the universe, how life began – and potential "neatly fills all the gaps". Which, for most scientists, is exactly the problem with ideas such as Birnbaum's. They make no testable predictions. Try to account for everything, and the risk is that you end up saying nothing.

And yet it is difficult not to feel some admiration. At least Birnbaum continues to ask the big questions. His immunity to mockery is impressive in its way. (That "magisterial" quote from the Warwick mathematician, Hugo van den Berg, turns out to come from an obviously sarcastic critique of the book; Birnbaum had simply plucked out the good bits.) At one point, I suggested to Birnbaum that there was a parallel between his jeweller's eye for beauty and his love of elegance in ideas. He eagerly agreed. His theory "is aesthetically elegant… and I like things aesthetically elegant." What got the professionals into trouble, he said, was over-thinking: "Everyone's missing it because they try too hard. You get there by relaxing. Letting go. Potential, possibility: it's the gentlest of all concepts." He half-closed his eyes. "Possibility is driving everything. It's so simple!"

Of course, he wasn't going to get everyone on board immediately. He'd been working at this for 30 years; it made no sense to give up now. "I don't have a degree in philosophy, I'm not an academic," he said. "There's only one saving grace to this book: it might be right."