In the garden of a terraced house in Oxford, on the hottest, sunniest day of the year, I meet Professor James R Flynn, an American-born academic who is a hero to many people. More than 30 years ago, he discovered a phenomenon that revolutionised the study of IQ and seemed finally to settle the argument over nature versus nurture. He showed that, across the world, average IQs had risen by roughly three percentage points every decade since at least 1930, and probably much longer.

Since evolution doesn’t work fast enough to produce genetic upgrading on that scale, it seemed that environment must be the dominant influence. According to Flynn, rising IQs went hand-in-hand with modernisation, which involves more years of education and more jobs that require analytic abilities and abstract thinking. The belief that better schooling, and positive discrimination in favour of disadvantaged children, could make a difference was seemingly vindicated.

The view, put forward by a number of British and American academics at that time, that black people’s IQs were genetically inferior to those of whites and Asians was finally discredited. So was the idea that African countries were poor because their inhabitants were stupid. IQs in developing countries also rise as they modernise and will eventually catch up those in developed countries. Best of all, rising IQs led to better moral reasoning, putting racism and sexism on the defensive.

Nearly all psychologists now accept what they call “the Flynn effect”, a remarkable accolade for a man who isn’t even a specialist in psychology, his academic subject being political studies. This is not the only reason to admire “Jim”, as family and friends call him. He is also an ardent democratic socialist who left an academic career in the US because he believed he was held back by his political views and his activity in the civil rights movement. He exiled himself to New Zealand in the 1960s, becoming a professor at the University of Otago in Dunedin. There, he helped found the Alliance party, and stood unsuccessfully for parliament against the local version of New Labour. He named his son, Victor, now an Oxford maths professor, after Eugene Victor Debs, five times presidential candidate for the Socialist Party of America.

This, I think, as we sit in Victor’s garden, is like meeting Einstein or Freud, men who changed for ever the way we see the world. Flynn, now 82, speaks fluently and wittily in Irish-American cadences (his grandfather, a teacher, fled the 19th-century Great Hunger in Ireland). He runs twice a day “though not as fast as I did 10 years ago”. He still teaches four-fifths of a timetable at Otago and writes nearly a book a year.

We are here to talk about his latest book, Does Your Family Make You Smarter? Having explored IQ differences between generations, he has turned his attention to those within generations, which is a different matter. It is already evident to me, after reading the book, that the Flynn effect doesn’t settle as much as some of us thought or hoped it did. And that by 21st-century standards, perhaps Flynn doesn’t quite measure up as a liberal hero.

The answer to the question in the title, Flynn explains, is that your family environment’s effect on your IQ almost disappears by the age of 17. An important exception is in the vocabulary component of IQ tests, where the effect persists into the mid-20s and can make a big difference, at least in the US, to the chances of getting into a top university. The home has most influence in early childhood but is swamped by later environments at school, university and work. And they will more closely match your genes because you will seek out (and be chosen for) environments that match your “genetic potential”, whether it’s basketball, carpentry or mathematics.

So having taken genes out of the equation, Flynn has apparently put them back in. Nurture hasn’t won after all. Over the years, a modernised environment will raise everyone’s game. But within each generation, the gap between groups – classes, races, genders – can, at least theoretically, remain the same as they were. In practice, groups previously excluded from full access to education and professional careers will close the gap.

Flynn emphasises that a rise in the potency of genes isn’t matched by a corresponding decline in the potency of environment: “The potency of one is added to the potency of the other.” Moreover, 20% of IQ differences are attributable to neither environment nor genes but to “chance factors” of which our ability to improve ourselves is the most important.

“You can be at the 98th percentile [more intelligent than all but 2% of the population],” he says. “At school, you kick teachers in the ankles, don’t hand in your homework, get into fights and end up being suspended. You become a bricklayer in a humdrum environment and it costs you, say, 10 IQ points. Then you think: to hell with this, all these guys talk about is girls and football, I’m going to university. Your IQ could rise from 120 to 130 and you’ve leapfrogged four-fifths of the people ahead of you.”

We are talking averages, of course. So I ignore the implied slur on bricklayers, accept the good news – that free will has a significant role – and move on.

If we are all getting brighter and better at moral reasoning, how does Flynn account for Donald Trump? “The rise of visual culture means far fewer people read serious novels and history. They live in a bubble of the present, believing what they are told because they have nothing to position it against. Improved analytic abilities do not make you a better citizen.”

And if we bring back grammar schools, is it possible to set a tutor-proof 11-plus exam? “Yes, but only by giving everyone a tutor.”

I have many more questions but one in particular looms over discussions about IQ and we both know we can’t avoid it. It was, after all, to challenge the late Arthur Jensen, professor of educational psychology at the University of California, Berkeley – who claimed the genes of African Americans were responsible for their inferior IQ scores – that Flynn began to examine the evidence on intelligence. But a sentence from his new book is nagging away at me. American blacks, it says, “come from a cognitively restricted subculture”.

This is hugely sensitive territory because, while it may be good to say genes don’t make people stupid, it isn’t so good to tell anyone their way of life does. Flynn, however, makes no apologies. “It’s whites, not blacks, who complain,” he says. “Blacks know the score. Facts are facts.” On recorded IQ tests, he says, African Americans have persistently lagged behind [pdf] most other ethnicities in America [pdf] (including, according to some commentators, black immigrants from, for example, the Caribbean) and this cannot be explained by the Flynn effect since, as he puts it, “blacks don’t live in a time warp”.

He then tells what sounds like a version of those dodgy jokes about the Irishman, the Scotsman and the Englishman. Except this isn’t a joke. “Go to the American suburbs one evening,” says Flynn, “and find three professors. The Chinese professor’s kids immediately do their homework. The Jewish professor’s kids have to be yelled at. The black professor says: ‘Why don’t we go out and shoot a few baskets?’”

As I emit a liberal gasp, he continues: “The parenting is worse in black homes, even when you equate them for socio-economic status. In the late 1970s, an experiment took 46 black adoptees and gave half to black professional families and half to white professionals with all the mothers having 16 years of education. When their IQs were tested at eight-and-a-half, the white-raised kids were 13.5 IQ points ahead. The mothers were asked to do problem-solving with their children. Universally, the blacks were impatient, the whites encouraging. Immediate achievement is rewarded in black subculture but not long-term achievement where you have to forgo immediate gratification.”

He tells me of research showing that “when American troops occupied Germany at the end of the second world war, black soldiers left behind half-black children and white soldiers left behind all-white. By 11, the two groups had identical average IQs. In Germany, there was no black subculture.”

Flynn refuses to speculate about the lingering effects of slavery and subsequent discrimination that have prevented African Americans from entering colleges and professional careers. Universities, he thinks, should do more research on racial differences and a new version of that 1970s study. “I have shown – this wicked person who actually looks at the evidence – that blacks gained 5.5 IQ points on whites between 1972 and 2002. There’s been no changes in family structure [the incidence of single-parent families], no gains in income. I suspect it’s an improvement in parenting. But I can’t prove it.”

I leave that sunlit garden in a troubled frame of mind. Flynn has made a great contribution to human knowledge and understanding. But he hasn’t settled the nature-against-nurture debate – and I wonder if he is now muddying the waters, constructing theories about parenting from flimsy evidence.