Controversy delights Dick Swaab; brains delight him; complexity delights him, though I don't know if you'd get that from reading his book, We Are Our Brains, in which causal links are made quite casually, like a man doing a crossword with a pencil.

The Dutch neurologist is, after a 50-year career, a giant in the field. He is a professor of neurobiology at the University of Amsterdam. His directorship of the Dutch Institute for Brain Research yielded material that has been sent to 500 other research groups in 25 countries. He has propounded groundbreaking theories in his specialist area: the impact on brain development in the womb. Nonetheless, his book, despite directing itself squarely to the layperson, has been miles more successful than he thought, selling 100,000 copies ("the publishers say they knew it would be a hit. But at the start, they only printed 3,000 copies. So I know that is not true."). There are a number of lines you might file under, "Well, there's a curiosity" (for instance: "In professional violinists, the part of the cerebral cortex that directs the fingers of the left hand is five times as large as it is in people who don't play a stringed instrument"). And yet the real fireworks of the book are both more predictable and more profound: Swaab says hormones and chemical substances in utero affect the development of our sexual orientation or, put more simply, you have a gay brain by the time you are born. Male and female brains have "hundreds of differences", which explain all the ways in which men and women are different; "phobia, impulsiveness, ADHD and depression later in life" can be traced back to a mother's fearfulness during pregnancy, which activates her baby's "fear axis".

It has become fashionable to use the word "exhaustive" when what reviewers actually mean is "long", and this book has been thus described a number of times; in fact, it is not exhaustive. Differences between the sexes are stressed and constantly referred to, but not named or referenced. Very far-reaching statements are made – for instance, that children who are adopted between nought and two have average IQs of 100, while children adopted between two and six have average IQs of 80. But there's no footnote. Which children? From where? How many? Adopted by whom? For what reason? Is it possible that the ones who were adopted between two and six weren't chosen sooner because prospective parents could tell they had a learning difficulty?

And then there are Swaab's delightful manners, the smooth, twinkling charm of a man who has spent a life engaged in things that fascinate him, so that even though every answer is basically "sod off …", it is impossible not to like him. But it is not, ultimately, impossible to unshackle oneself from the confident steam train of his assertions.

"I had this problem," he reminisces. "When I found the first differences between male and female in the brain, I got attacked by the feministic movement. Because it was not allowed to have any differences, not in the brain. All differences in brain and behaviour were due to society." He says all this as though it's years since, and yet as a card-carrying feministic, I do not feel that our concerns have been totally addressed. For one, there is a circularity to the argument: a connection that is supposedly hard-wired, say that between a mother and a son, is illustrated by the fact that "a wounded soldier on a battlefield will always call for his mother, not his father". But this surely is social, because the mother will have almost certainly been the caregiver? We should at least wait until a generation has been raised equally by both parents, then mindlessly slaughter some, and then see who they call for, shouldn't we?

Hypotheses about raised testosterone in the womb due to maternal stress are offered on the basis that girls will need to be more like boys if they're to be born into a hostile environment, because they'll need to be more "robust and competitive". Yet, not long after, there's a joke about the long-discredited dominant mother theory (about homosexuality), in which Swaab says: "I made a habit over the years of asking the medical students I taught (250 at a time) which of them did not have a dominant mother. No one ever raised their hand." Sure, it's droll, but dominance, robustness, competitiveness, these are all the same essential trait – is he saying that women ultimately become men or is he saying that traits that are quintessentially male are also quintessentially female? If so, doesn't that de-sex them? Doesn't that make them "human"? He shrugs this off. "The problem often is that if scientists talk about something, it is on the basis of the population. Some women have very male types of behaviour. Some males have very feminine type of behaviour, but as a population they are different." This is the "standard-deviation" lecture – "you won't understand, because you're only a layperson." It didn't really address my question, which was, aren't you yourself assigning "maleness" and "femaleness" to traits that we actually all have? Rebecca Jordan-Young, author of Brainstorm: The Flaws in the Science of Sex Differences, argues that "Brain Organisation Theory is little more than an elaboration of longstanding folktales about antagonistic male and female essences".

Other old assumptions about men and women – for instance, that men are aroused by images of naked women, whereas women can't be aroused visually in the same way – are trotted out as fact. But there's new evidence that women respond very strongly to all kinds of erotic images, including copulating bonobos. This is based on measuring bloodflow to the vagina rather than an MRI scan. It makes this apparently "hard-wired" difference look like male wishful thinking. "Well," Swaab counters, "it's measuring bloodflow to the vagina. How did the blood flow to the vagina? By getting a message from the brain." "Then how do you account for the different results?" "Because the image is too small in the brain, you miss the small nuclei in the hypothalamus. It's a limitation of the imaging." (I was too polite to push this point. But that seems like a rum old game, publishing something as true, then recanting it so easily. You either believe that men and women respond differently or you don't.)

As much of a pain in the neck as a feministic can be, Swaab's most vocal opponents, 20 years ago, were gay men, after he identified a "gay brain". He makes the sound point, in the book and in person, that this was actually great for homosexuality – asserting sexual orientation to be innate ended all those painful discussions about whether to blame the parents, whether one could persuade or train or pray oneself out of being gay. "People can only live a happy life if they can live the way their brain has been programmed, and the state should accept that, and guarantee them the freedom to live that way so long as they don't harm others. You should have the freedom to live as a homosexual, a transsexual, a heterosexual, and be protected by the state," he says simply. And his discoveries about transsexuality had these direct political consequences; the science put sexuality on the human rights agenda, in Brussels and beyond. There are laws protecting gender reassignment that simply wouldn't exist without Swaab's findings.

And yet, his conclusions about the gay brain – specifically, that it has an enlarged biological clock (the suprachiasmatic nucleus) – have run into dispute. The American neuroscientist Simon LeVay remarked recently: "Dick Swaab's group wrote in 1990 that a small group called the suprachiasmatic nucleus was larger in gay men than in heterosexual men. The suprachiasmatic nucleus is involved in the regulation of circadian rhythms, not sex. Swaab's report has not been confirmed by other groups. If the finding is correct, it is unclear whether it is meaningful."

I put this to Swaab, that neither LeVay nor anyone else had managed to reproduce his findings. He sort of waved at me, as if I was making an annoying noise. "The important thing is that it's a marker for something that happened earlier in your brain development. Of course the clock is very much linked to sexual behaviour. There are graphs of students that don't have anything else to do, but if you look to their sexual behaviour, it goes up at 10, 11, 12. There is a very clear day/night correlation in sexual behaviour. There are more nightclubs than dayclubs."

"This is absolutely true," I say. Thinking: "Seriously? Results that can't be replicated, called into question anyway because they were mainly the brains of people who had died of Aids ('We controlled for that! Because many heterosexual people at that time were dying of Aids also. So we were able to see that the enlargement hadn't affected them'), we're going to take on trust because of when nightclubs are open?"

Other times, his benign dismissiveness can be quite bracing (talking about the Dutch Hunger study, I put it to him that the Leningrad Siege study, also conducted on babies born to women who were starving, in the same time period, in the same sorts of numbers, had findings that were basically opposite. "This is not a good study," he replies. "If this were a good study, I would have heard of it." He is endearingly mischievous – at one point, explaining that anorexia was down to the brain's way of coping with glucose, which is why anorexics often have problem births; he utterly rejected the idea that it was anything to do with cultural imagery. "Fashion selects girls with anorexia and bulimia, because they look better. But that doesn't mean that fashion is causing it. It is merely using it." On the subject of mortality, and of his own brain, he is absolutely wonderful. "Some time ago, I went through a difficult period in terms of health and made some notes for my colleagues, 'Mind this', 'Don't do that'. And then I survived. I wanted to keep an eye on this and that, because I know my own character and I wanted to know the relationships between structure and function."

What characteristics did you want to investigate? "I don't tell you. You will hear in good time." The Netherlands already has laws on euthanasia that astonished me – you can apply to die for long-term mental illnesses, schizophrenia, lifelong depression. But Swaab is keen to see some provision made for people who aren't ill, who are just old. "They have pains, they don't want to live any more. It was nice, but not any more. It doesn't fit into the law in the Netherlands that was made for diseases, not for the end of a life that has been accomplished." "This is a campaign of yours?" "Yes," he says, beaming. "This is my short-term solution, and my long-term solution."

Ultimately, though, I keep returning to Jordan-Young's critique: "I think we have to start asking different questions. It isn't possible to do experiments on the causes of gender or sexuality, and we can never really go backwards and separate 'biological' from 'social' elements – that is trying to divide what is actually indivisible. So, I think that we might do better to focus on understanding plasticity and ongoing development, and also to stop obsessing so much about sex differences." Many of Dick Swaab's boldest statements turn out, on closer examination, to be an MRI scan of chauvinism. It sounds interesting, but you can't read proper meaning into it; that part of the brain is too small.

To order We Are Our Brains: From the Womb to Alzheimer's by D F Swaab (Allen Lane), for £16.00 with free UK p&p, go to theguardian.com/bookshop or call 0330 333 6846.