In 1976 the National Council for Civil Liberties, the respectable (and responsible) pressure group now known as Liberty, made a submission to parliament's criminal law revision committee. It caused barely a ripple. "Childhood sexual experiences, willingly engaged in with an adult," it read, "result in no identifiable damage … The real need is a change in the attitude which assumes that all cases of paedophilia result in lasting damage."

It is difficult today, after the public firestorm unleashed by revelations about Jimmy Savile and the host of child abuse allegations they have triggered, to imagine any mainstream group making anything like such a claim. But if it is shocking to realise how dramatically attitudes to paedophilia have changed in just three decades, it is even more surprising to discover how little agreement there is even now among those who are considered experts on the subject.

A liberal professor of psychology who studied in the late 1970s will see things very differently from someone working in child protection, or with convicted sex offenders. There is, astonishingly, not even a full academic consensus on whether consensual paedophilic relations necessarily cause harm.

So what, then, do we know? A paedophile is someone who has a primary or exclusive sexual interest in prepubescent children. Savile appears to have been primarily an ephebophile, defined as someone who has a similar preferential attraction to adolescents, though there have been claims one of his victims was aged eight.

But not all paedophiles are child molesters, and vice versa: by no means every paedophile acts on his impulses, and many people who sexually abuse children are not exclusively or primarily sexually attracted to them. In fact, "true" paedophiles are estimated by some experts to account for only 20% of sexual abusers. Nor are paedophiles necessarily violent: no firm links have so far been established between paedophilia and aggressive or psychotic symptoms. Psychologist Glenn Wilson, co-author of The Child-Lovers: a Study of Paedophiles in Society, argues that "The majority of paedophiles, however socially inappropriate, seem to be gentle and rational."

Legal definitions of paedophilia, needless to say, have no truck with such niceties, focusing on the offence, not the offender. The Sex Offenders Act 1997 defined paedophilia as a sexual relationship between an adult over 18 and a child below 16.

There is much more we don't know, including how many paedophiles there are: 1-2% of men is a widely accepted figure, but Sarah Goode, honorary research fellow at the University of Winchester and author of two major 2009 and 2011 sociological studies on paedophilia in society, says the best current estimate – based on possibly flawed science – is that "one in five of all adult men are, to some degree, capable of being sexually aroused by children". Even less is known about female paedophiles, thought to be responsible for maybe 5% of abuse against pre-pubescent children in the UK.

Debate still rages, too, about the clinical definition of paedophilia. Down the years, the American Psychiatric Association's Diagnostic and Statistical Manual of Mental Disorders – "the psychiatrist's bible" – has variously classified it as a sexual deviation, a sociopathic condition and a non-psychotic medical disorder. And few agree about what causes it. Is paedophilia innate or acquired? Research at the sexual behaviours clinic of Canada's Centre for Addiction and Mental Health suggests paedophiles' IQs are, on average, 10% lower than those of sex offenders who had abused adults, and that paedophiles are significantly less likely to be right-handed than the rest of the population, suggesting a link to brain development. MRI scans reveal a possible issue with paedophiles' "white matter": the signals connecting different areas of the brain. Paedophiles may be wired differently.

This is radical stuff. But there is a growing conviction, notably in Canada, that paedophilia should probably be classified as a distinct sexual orientation, like heterosexuality or homosexuality. Two eminent researchers testified to that effect to a Canadian parliamentary commission last year, and the Harvard Mental Health Letter of July 2010 stated baldly that paedophilia "is a sexual orientation" and therefore "unlikely to change".

Child protection agencies and many who work with sex offenders dislike this. "Broadly speaking, in the world of people who work with sex offenders here, [paedophilia] is learned behaviour," says Donald Findlater, director of research and development at the Lucy Faithfull Foundation, a charity dedicated to preventing child sexual abuse, and, before it closed, manager of leading treatment centre the Wolvercote Clinic. "There may be some vulnerabilities that could be genetic, but normally there are some significant events in a person's life, a sexually abusive event, a bullying environment … I believe it is learned, and can be unlearned."

Chris Wilson of Circles UK, which helps released offenders, also rejects the idea that paedophilia is a sexual orientation: "The roots of that desire for sex with a child lie in dysfunctional psychological issues to do with power, control, anger, emotional loneliness, isolation."

If the complexity and divergence of professional opinion may have helped create today's panic around paedophilia, a media obsession with the subject has done more: a sustained hue and cry exemplified by the News of the World's notorious "name and shame" campaign in 2000, which brought mobs on to the streets to demonstrate against the presence of shadowy monsters in their midst. As a result, paranoia about the danger from solitary, predatory deviants far outweighs the infinitely more real menace of abuse within the home or extended circle. "The vast majority of sexual violence is committed by people known to the victim," stresses Kieran Mccartan, senior lecturer in criminology at the University of the West of England. Only very rarely is the danger from the "stranger in the white van", Mccartan says.

The reclassification of paedophilia as a sexual orientation would, however, play into what Goode calls "the sexual liberation discourse", which has existed since the 1970s. "There are a lot of people," she says, "who say: we outlawed homosexuality, and we were wrong. Perhaps we're wrong about paedophilia."

Social perceptions do change. Child brides were once the norm; in the late 16th century the age of consent in England was 10. More recently, campaigning organisations of the 70s and 80s such as the Paedophile Information Exchange (PIE) and Paedophile Action for Liberation were active members of the NCCL when it made its parliamentary submission questioning the lasting damage caused by consensual paedophilic relations.

Even now there is no academic consensus on that fundamental question – as Goode found. Some academics do not dispute the view of Tom O'Carroll, a former chairman of PIE and tireless paedophilia advocate with a conviction for distributing indecent photographs of children following a sting operation, that society's outrage at paedophilic relationships is essentially emotional, irrational, and not justified by science. "It is the quality of the relationship that matters," O'Carroll insists. "If there's no bullying, no coercion, no abuse of power, if the child enters into the relationship voluntarily … the evidence shows there need be no harm."

This is not, obviously, a widely held view. Mccartan uses O'Carroll's book Paedophilia: the Radical Case in his teaching as "it shows how sex offenders justify themselves". Findlater says the notion that a seven-year-old can make an informed choice for consensual sex with an adult is "just preposterous. It is adults exploiting children." Goode says simply: "Children are not developmentally ready for adult sexuality," adding that it is "intrusive behaviour that violates the child's emerging self-identity" and can be similar in long-term impact to adults experiencing domestic violence or torture.

But not all experts are sure. A Dutch study published in 1987 found that a sample of boys in paedophilic relationships felt positively about them. And a major if still controversial 1998-2000 meta-study suggests – as J Michael Bailey of Northwestern University, Chicago, says – that such relationships, entered into voluntarily, are "nearly uncorrelated with undesirable outcomes".

Most people find that idea impossible. But writing last year in the peer-reviewed Archives of Sexual Behaviour, Bailey said that while he also found the notion "disturbing", he was forced to recognise that "persuasive evidence for the harmfulness of paedophilic relationships does not yet exist".

If that assertion does nothing else, it underlines the need for more research on paedophilia – something on which everyone in the field at least is agreed. There is, too, broad consensus around the idea that the approach to paedophilia must be about management and prevention: on stopping potential offenders making that contact (or downloading that image).

Initiatives such as Stop It Now!, which Findlater runs, exemplify this: a telephone helpline offering advice to people worried they may be having inappropriate sexual impulses. A similar German programme, Prevention Project Dunkelfeld, has as its slogan: "You are not guilty because of your sexual desire, but you are responsible for your sexual behaviour. There is help."

For convicted abusers, Circles UK aims to prevent reoffending by forming volunteer "circles of support and accountability" around recently released offenders, reducing isolation and emotional loneliness and providing practical help. In Canada, where it originated, it has cut reoffending by 70%, and is yielding excellent results here too. The goal of all treatment, Findlater says, is "people achieving a daily motivation not to cause harm again. Our goal is self-management in the future."

For Goode, though, broader, societal change is needed. "Adult sexual attraction to children is part of the continuum of human sexuality; it's not something we can eliminate," she says. "If we can talk about this rationally – acknowledge that yes, men do get sexually attracted to children, but no, they don't have to act on it – we can maybe avoid the hysteria. We won't label paedophiles monsters; it won't be taboo to see and name what is happening in front of us."

We can help keep children safe, Goode argues, "by allowing paedophiles to be ordinary members of society, with moral standards like everyone else", and by "respecting and valuing those paedophiles who choose self-restraint". Only then will men tempted to abuse children "be able to be honest about their feelings, and perhaps find people around them who could support them and challenge their behaviour before children get harmed".

• This article was amended on 3 January 2012. The original incorrectly suggested that the Diagnostic and Statistical Manual of Mental Disorders was published by the American Psychological Association, and misspelled Dunkelfeld as Dunkenfeld. This article was further amended on 21 January 2013 because the original referred to Sarah Goode as a senior lecturer at the University of Winchester. This has been corrected to say honorary research fellow.