Medical research on animals that contain material from humans – such as brain cells – should be more tightly regulated, according to a report from a leading panel of scientists and ethicists.

The report, which included a public consultation, says such research needs more scrutiny and clearer legal boundaries.

Scientists already use animals that contain human material in work to understand medical conditions such as Down's syndrome. But the report's authors said politicians should devise regulations to cover likely advances.

The team highlighted future research into human cognition and reproduction as areas of greatest concern to the public.

Martin Bobrow, professor of medical genetics at the University of Cambridge, led the panel set up by the Academy of Medical Sciences. He said: "Where people worry is when you get to the brain, the germ cells and the sentinel features that help people recognise what is a person, as opposed to a rat or a rabbit.

"Things like skin texture, facial shape, speech, replacing brain cells with human cells, allowing the development of human germ cells in animals. And particularly where there is any possibility of fertilisation within an animal."

He said the public was also concerned about animals whose appearance was deeply disturbing.

The professor said the panel was not recommending such work be banned outright, but that these were areas where the value of the science should be most carefully considered. Current laws around the use of animals in scientific research would cover most eventualities for now but these rules would not be enough for the techniques of the future, he said.

One "animal model" that includes human material is the Down's syndrome mouse. It carries a copy of human chromosome 21 among its DNA. Using this model, scientists are gaining insight into the condition.

"Changing animals by putting human genes or cells into their structure is one way of making them more resemble the bit of the human condition you're interested in studying," said Bobrow.

Tom Baldwin, a philosopher at the University of York and panel member, said ethical decisions might need to be based on the "great ape test", referring to the UK moratorium against using humans' closest animal relatives in medical research.

"If you start putting very large numbers of human brain cells into primates, suddenly you might transform primates into something that has some of the capacities that we regard as distinctively human – speech or other ways of being able to manipulate or relate to a human," he said. "These possibilities, at the moment, are largely being explored in fiction but we need to start thinking about them now."

The report recommends that a science and ethics committee with input from the public be set up to assess the merit of scientific projects in sensitive areas.

Lovell-Badge said continuing public discussion about the issue was key, "so that those sorts of experiments are discussed openly. Some of them certainly should be done, but it needs to be done in an open way."

Paul Nurse, president of the Royal Society, said this sort of research required open discussion to ensure it moved in a direction that the general public remained comfortable with. "The Royal Society is supportive of the recommendations made in this report, especially the call for a national expert body … Proper scrutiny and regulation of this developing field now will ensure that society benefits from its advances fully."