Friday's report by the Academy of Medical Sciences on the increasingly fuzzy boundaries between the human and the animal is the latest in a long series of policy reflections on how to keep pace with developments in the biosciences.

It can justly be said that politics and regulation have not dealt well with our newfound capacities for muddying the boundaries between us and other species. And yet the last two decades have witnessed an unprecedented growth in bioscientific techniques that increasingly call into question what it means to be human. Take the human genome project: many of us may have intuitively suspected that we might have more genetically in common with the chimpanzee than even Darwin had envisaged, only then to be told of our cousinly closeness to the fruit fly, maize and the zebra fish.

Casting a glance back to the 1990s, trans-species transplantation seemed to promise a new era of limitless animal organs and tissues. Who knows, it may still. But that dream slowly sank from view amid concerns about potentially catastrophic trans-species disease, and increasing evidence of its poor performance in preclinical trials with primates. Move forward a decade and we have the trans-species embryo debate, resulting in legislative changes permitting a whole new class of research embryos incorporating animal DNA. So to the classical question of "what is an embryo", has been added the equally vexing puzzle "what is an animal".

Bioscientific hybrids are difficult to categorise, disorderly, existing on the fringes of the humanised animal and the animalised human. And yet policymaking has arguably had a poor track in getting to grips with and understanding trans-species innovation. Trans-species biologies present acute difficulties especially in terms of regulation because they confuse and traverse regulatory institutional boundaries.

In the UK, as elsewhere, regulatory agencies have tended to regulate humans on the one hand, and animals on the other, with little consideration for what might lie between. The tendency has been to deal with all things animal through the Home Office and its Animal Procedures Inspectorate, and to deal with all things human through the Department of Health. There are good and disturbing grounds for suspecting this division has become increasingly naive and meaningless, as the biosciences enter their trans-species future.

In the late 1990s the UK deemed it necessary to establish a regulatory body to manage the many murky trans-species hybrid implications of xenotransplantation, the UK Xenotransplantation Interim Regulatory Authority. But far from being a porous conduit between the DoH and the Home Office, UKXIRA found itself hamstrung. The Home Office would seek its advice on animal experiments involving primates but would not allow the authority to see confidential trial applications or the results of previous studies. This proved to be a poor basis on which to advise the DoH about the wisdom or otherwise of proceeding to clinical trials with humans. UKXIRA wasn't perfect, but it represented an important attempt to overcome the regulatory divide between the human and the animal. The government's decision to disband UKXIRA in 2006 could justifiably be viewed as myopic and short-sighted, given the trans-species direction of travel in the biosciences. In losing UKXIRA, the UK also lost important institutional experience and a model for dealing with interspecies biotechnological developments.

Even more recently, the UK trans-species embryo debate points to equally serious flaws in the regulation of the wild indeterminate zones between us and other animals. One strategy evident in the run-up to changes in legislation allowing the creation of trans-species embryos was to downplay that they might be trans-species at all. Just reflect for a moment on shifts in the language used to describe these embryos: the DoH, for instance, started out talking about "trans-species embryos" before finally settling on its preferred term, "human admixed embryos". In other words, these embryos might be a bit mixed up, but they're essentially human. No worries.

Such embryos would allow stem-cell scientists to use animal eggs rather than scarce human eggs to create stem-cell lines. The animal nuclei could be removed and replaced with human nuclei leaving only a residue of animal egg DNA behind. It was striking to see this process now described by some stem-cell scientists as "especiation" in place of the more scientifically conventional term "enucleation". In this way, the vexing animal is tidied away behind a thin veneer of language and rhetoric.

More worrying is the continuing confusion over whether trans-species embryos should be regulated by human or animal agencies – again, the Home Office or the DoH. That boundary comes down to potentially confused assessments of whether it is the animal or the human which "predominates" in the resulting embryo. I had the privilege of discussing this recently with an eminent UK reproductive scientist who had been involved in crafting the legislation. "Ultimately," he said, "it has to be either human or animal to be regulated ... otherwise we would have to do away with our whole regulatory edifice." Well, that is exactly what we may have to do.

It could be argued that the report by the Academy of Medical Sciences searchingly arrives at a point of dissatisfaction with the bipedal and binary regulation of transbiology. Perhaps it's time for an overhaul of our institutions, their language and assumptions about what it is to be human, animal and the many murky zones in between.