On 19 October 1983, US physicist Willy Fowler received a phone call that most scientists can only dream of. In careful tones, a member of the Swedish Academy of Sciences told him he had been awarded that year's Nobel prize for physics. His award, he was informed, had been given for his research that had helped reveal the stellar origins of the elements from which our bodies, solar system and universe are made.

Then came the shock. Fowler, who was 72 years old at the time, was told he would share the prize with Indian astrophysicist Subrahmanyan Chandrasekhar, who had carried out pioneering work on the structure of stars. Of Fowler's own close collaborator, Fred Hoyle – the British scientist who had led their joint research work – there was no mention.

The American physicist was stunned, he later admitted. And so were other scientists, men and women who remain puzzled by the omission to this day. "I have no idea how the Swedes decided to make an award to Chandrasekhar and Fowler but not to Hoyle," admits astronomer Lord Rees, president of the Royal Society. "However, I think it would be widely accepted that it was an unfair misjudgment."

In 1957, Hoyle and Fowler showed that all the elements from which our world is made – from carbon atoms to uranium atoms – had been cooked inside stars eons ago from a basic fuel of hydrogen. These heavy elements were then blasted into space in great stellar explosions called supernovae, where they later congealed into planets, mountains – and humans. We are stardust, in other words.

"There is no doubt that Fowler and Hoyle's 1957 paper is of Nobel quality and standing," says Hoyle's biographer, Simon Mitton, a Cambridge astronomer. "And in terms of understanding the chemical elements, Hoyle made the greater contribution when you compare it with Fowler's. The latter was a nuclear physicist who provided basic data. Hoyle provided the insights."

Scientists' dismay at the refusal to give Hoyle a Nobel prize is understandable, although it should be noted that he could be cantankerous and opinionated and had offended a large number of influential colleagues unused to his Yorkshire bluntness. He had called some of them liars and cheats in public, while his beliefs, in later life, verged on the lunatic. He said Earth was being constantly bombarded by microbes from outer space and that these were responsible for outbreaks of flu and other illnesses. He also claimed that remains of archaeopteryx – the British Museum fossil that demonstrates the early link between dinosaurs and birds – was a fake. Such notions went down badly in scientific circles.

But were they grounds for refusing Hoyle a Nobel prize? Understanding the origin of the elements was a major intellectual breakthrough. Who cares if he was a bit fruity about flu and fossils? And surely by awarding one scientist a Nobel prize for a piece of work while refusing to give it to the senior partner in the effort, the Swedish academy was being deliberately provocative? Was this a warning to scientists about the dangers of speaking out of turn?

Such issues raise timely questions about the nature of the Nobel prize. This week, the winners of the 2010 science Nobels will be revealed, with the announcement due tomorrow of the physiology prize. Those elevated will no doubt bask in some well-deserved publicity. Others will complain of major errors. "Every year I expect Stephen Hawking to be chosen and every year I am disappointed," says renowned theoretical physicist Freeman Dyson.

In the background, there is a growing feeling among senior scientists that the Nobels, which are now in their 110th year, need to change fundamentally. For example, in an era in which so much scientific research is generated by vast collaborative teams, is it right to limit each annual award to its current maximum of three? Already a row is brewing over who should be honoured with a Nobel if physicists finally discover the elusive Higgs boson, one of the main targets of the Large Hadron Collider at Cern near Geneva.

The Higgs is predicted to exist as part of a mechanism that gives other particles their mass, a theory developed independently by three groups in 1964. Two physicists in Brussels, François Englert and Robert Brout, were the first into print, followed by Peter Higgs at Edinburgh University. The third paper came from two US physicists, Gerald Guralnik and Carl Hagen, and Tom Kibble in London. The arithmetic leaves six men in contention for an award that can go to three people at most. Even before the Higgs particle has been discovered, tensions have reached a head, with the US physicists suspecting a European conspiracy to write their work out of history.

This point is stressed by Lord Rees. "The limit of three winners – a constraint that could surely be changed – leads to two problems. Sometimes, a major advance is a team effort and it would be better to recognise the team than just a maximum of three individuals. Sometimes, there are several people who have worked independently on a topic and it is then invidious to pick out just three."

Then there is the limit of physics, physiology (strictly physiology and medicine) and chemistry as individual topics for Nobel prizes in science. Should this total not be expanded to reflect the diversity of 21st-century research? This point was raised recently by a panel of distinguished scientists, including UK Nobel winner and cancer expert Tim Hunt, who suggested there should be Nobels for environmental science and climate change; research into public health; plant science; and evolutionary biology.

However, the most pressing issue is a simple one: what exactly should a Nobel prize be awarded for? Should it be given for a major piece of insightful work or should it reflect a scientist's standing and overall achievements? In other words, to what extent should science tolerate the maverick and to what degree does the Nobel committee need to protect the image of its "brand"? No one example better personifies these issues than that of Hoyle.

Fred Hoyle was the son of a cloth merchant from Bingley. He revealed a prodigious intellect from early childhood and after leaving grammar school studied at Emmanuel College, Cambridge, becoming the university's Plumian professor of astronomy. There, in the 60s, he championed the cause of the steady state theory, which held that the universe had been in constant expansion for eternity. By contrast, supporters of the rival big bang theory argued that the universe had exploded into existence in a single event at some point in the finite past. Then, in the fire of this eruption, the elements were cooked together from basic particles "in less time than it takes to cook a dish of duck and roast potatoes," according to the big bang's main proponent, George Gamow.

In the end, measurements of cosmic radiation showed the latter idea to be correct – but not completely. Careful analysis also showed the elements could not have been cooked in the big bang's first seconds as had been thought. This left a mystery around the origin of the elements. And it is here that Hoyle made his mark. He had already developed the idea of nucleosynthesis as a key component of the steady state theory. Inside stars, under colossal pressures and temperatures, hydrogen nuclei fused to form nuclei of helium, he argued. Then, helium nuclei combined to form beryllium, and so on until carbon, oxygen, iron, silicon and other heavy elements were created.

It was a nifty idea. But there was a catch. For the idea to work, Hoyle calculated that inside stars carbon would have to exist in a very special state: the 7.65 MeV state of carbon-12. Without it, nucleosynthesis could not proceed beyond a very simple stage. However, no one had ever observed carbon in this state.

None the less, Hoyle insisted it must exist and this, says Marcus Chown in his book The Magic Furnace, was simply "the most outrageous prediction" ever made in science. "If [the 7.65 MeV state] did not exist, Hoyle reasoned, the universe would contain no carbon. And if there was no carbon, there would be no human beings. Thus Hoyle was saying – and nobody had ever used logic as outrageous as this before – that the mere fact he was alive and pondering the question of carbon was proof the 7.65 MeV state existed."

With Fowler's help, Hoyle did indeed find the 7.65 MeV state and the pair, working with astronomers Margaret and Geoffrey Burbidge, wrote "Synthesis of the Elements in Stars" for the Review of Modern Physics. Despite its four authors, the paper is principally Hoyle's work. "To this day, he is the only person to have made a successful prediction from an anthropic argument in advance of an experiment," adds Chown.

In effect, Hoyle, a highly imaginative man whose works included science-fiction classic A for Andromeda, was saying: I am, therefore I am right, an intriguing argument to say the least and, given its successful outcome, it was surely worthy of a Nobel. So why did Hoyle not get one? Without access to the Swedish academy's minutes, it is impossible to provide an exact reason, but Mitton is convinced Hoyle's previous run-ins with the Nobel prize committee played a crucial role. In 1974, British radio astronomer Antony Hewish had been awarded a Nobel for his work in discovering the first pulsar – a rotating neutron star. But his student, Jocelyn Bell Burnell, had not been recognised, despite the fact she was first to notice the stellar radio source that was later realised to be a pulsar. Hoyle accused Hewish of stealing her data, a remark that made headlines round the world and left Hoyle facing a libel suit. So he wrote a grovelling letter to the Times, blaming – instead – the Nobel prize committee for the mistake. "He was so critical of the committee that I imagine someone there just took a large pen and crossed his name off the list of those being considered for future prizes," says Mitton.

Seen from this perspective, Hoyle was the victim of his own intemperate nature, while the Nobel prize committee was guilty of a petty lack of objectivity. But there are other ways of looking at the issue, says British scientist Sir Harry Kroto, winner of the 1996 Nobel prize for chemistry. A Nobel is not just an award for a piece of work but is a recognition of a scientist's overall reputation, he believes. And by that definition, Hoyle – who died in 2001, never having recanted his belief in the steady state theory even when it was clearly demonstrated to be wrong – was unworthy of a prize.

"Apart from his earlier work, his arrogant, misplaced assumption of his own genius together with his blunt northern stubbornness, of which he was so proud, caused him to be wrong so often on high-profile issues that people have forgotten when he was right," says Sir Harry. "The Nobel committees go to inordinate lengths to do the best they can and in this case I think they thought Hoyle was so arrogant and dismissive of others that he would use the prestige of the Nobel prize to foist his other truly ridiculous ideas on the lay public. The whole scientific community felt that."

In short, science matters more than the individual. Hence the sacrifice of Fred Hoyle.