Scientists in the US claim to have used a bone marrow transplant to cure mental illness in a study that could have profound implications for patients with psychiatric problems.

Bone marrow transplants are routinely used to treat leukaemia and other life-threatening diseases, but have never been used to treat mental health problems.

The team, led by a Nobel prizewinning geneticist, found that experimental transplants in mice cured them of a disorder in which they groom themselves so excessively they develop bare patches of skin. The condition is similar to a disorder in which people pull their hair out, called trichotillomania.

"A lot of people are going to find it amazing," said Mario Capecchi at the University of Utah, who won the Nobel prize for medicine in 2007 for his work on mouse genetics. "That's the surprise: bone marrow can correct a behavioural defect."

The team said their work is the first to reveal a direct link between a psychiatric disorder and faulty immune cells, which grow in bone marrow before moving to the brain to protect nerve cells from damage.

Capecchi said the condition the animals develop is comparable to obsessive-compulsive disorder (OCD), and could shed fresh light on the roots of the disorder. Other illnesses including depression, schizophrenia and autism might also be linked to problems with the immune system, he added.

"The recognition that many neuropsychiatric diseases have a direct connection to the immune system emphasises that we should be taking immune deficiencies associated with neuropsychiatric disease much more seriously," Capecchi told the Guardian "We know a lot more about the immune system and how to treat immune deficiencies than we know about how our brain works and what the drugs used to treat neuropsychiatric disorders are doing," he added.

The cost and health risks associated with bone marrow transplants are such that they would never be used to treat mental health problems in people, but the findings will inspire research into immune-based therapies for psychiatric disorders, according to scientists who were not involved with the work.

Naomi Fineberg, a consultant psychiatrist at Queen Elizabeth II Hospital in Welwyn Garden City, said the work was a major development. "This finding is clearly important in directing research in new directions for OCD and OCD-spectrum disorder treatments. Given the intransigence of OCD symptoms, and the fact that roughly one third of treated OCD patients fail to make a good recovery, new treatment directions in this field are sorely needed."

In the study, Capecchi's team bred mice that carried a mutation in a gene called Hoxb8 that causes faulty immune cells to grow in the bone marrow. Mice that carry the defective gene groom themselves too often and for too long, leaving them with bare patches and skin wounds.

Writing in the US journal Cell, the team describe how transplanting healthy bone marrow into the mice cured them of the grooming disorder. In later operations, the scientists induced the disorder in healthy mice by giving them bone marrow from affected mice.

"We're showing there is a direct relationship between a psychiatric disorder and the immune system, specifically cells named microglia that are derived from bone marrow," Capecchi said. There are two kinds of microglial cells in the brain. Around 60% form in the brain in the earliest stages of human development, while the remaining 40% originate in bone marrow and then move to the brain.

"This is immensely important and incredibly exciting. It's definitely something people will want to follow up," said Douglas Blackwood, professor of psychiatric genetics at Edinburgh University. "Current treatments for these kinds of conditions are not incredibly effective and there's a massive need for alternatives."

Other researchers were more cautious about the work. Paul Salkovskis, clinical director of the Maudsley Hospital Centre for Anxiety Disorders and Trauma in London, said it was impossible to draw strong conclusions about the role of the immune system in human mental illnesses from the study. "Excessive grooming in mice is not a good model for obsessive-compulsive disorder in humans, a condition that can be treated effectively with cognitive behavioural therapy," he said.