WHEN Roland Fryer was about 15, a friend asked him what he would be doing when he was 30. He said he would probably be dead. It was a reasonable prediction. At the time, he was hanging out with a gang and selling drugs on the side. Young black men in that line of work seldom live long. But Mr Fryer survived. At 30, he won tenure as an economics professor at Harvard. That was four months ago.

Mr Fryer's parents split up when he was very young. His father was a maths teacher who went off the rails: young Roland once had to borrow money to bail him out of jail. His great-aunt and great-uncle ran a crack business: young Roland would watch them cook cocaine powder into rocks of crack in a frying pan in the kitchen. Several of his relatives went to prison. But Mr Fryer backed away from a life of crime and won a sports scholarship to the University of Texas. He found he enjoyed studying, and was rather good at it. By the time he was 25, the president of Harvard was hectoring him to join the faculty.

Mr Fryer now applies his supple mind to the touchy, tangled issue of racial inequality. Why are African-Americans so much less prosperous than whites? Why do so many black children flounder in school? Why do so many young black men languish behind bars? Why are stories like Mr Fryer's considered so surprising?

Black and white Americans tend to produce different answers to these questions, and there is also heated disagreement within both groups. Some blacks think their glass is three-quarters full; others think it three-quarters empty. Optimists can point to obvious improvements. Little more than four decades ago, blacks in the South could not vote. This year, a black man may be elected president. Under segregation, southern blacks were barred from white schools, neighbourhoods and opportunities. Now, racial discrimination is both illegal and taboo. Blacks have pierced nearly every glass ceiling. The secretary of state, the boss of American Express and the country's most popular entertainer (Oprah Winfrey) are all black.

Life for the average African-American has also improved remarkably. The median black household income has risen from $22,300 (in 2006 dollars) in 1967 to $32,100 in 2006. Black life expectancy has soared from 34 in 1900 to 73 today. Most blacks today are middle class.

Yes, say the pessimists, but the gap between what blacks and whites earn and what they learn, which narrowed steadily between the 1940s and the late 1980s, has more or less frozen since then. Blacks' median household income is still only 63% of whites'. Academically, black children at 17 perform no better than a white 13-year-old. Blacks die, on average, five years earlier than whites. And though the black middle class has grown immensely, many blacks are still stuck in crime-scorched, nearly jobless ghettos.

What ails black America? Public debate falls between two poles. Some academics and most civil-rights activists stress the role played by racial discrimination. It may no longer be overt, they argue, but it is still widespread and severe. Julian Bond of the National Association for the Advancement of Coloured People reckons that racism is still “epidemic” in America.

Black conservatives, while never denying that racism persists, think it much less severe than before and no longer the main obstacle to black advancement. Bill Cosby, a veteran comedian, tours the country urging blacks to concentrate on improving themselves: to study hard, to work hard and—especially—to shun the culture of despair that grips the ghetto.

The debate is often bitter. Michael Eric Dyson, a leftish academic, argues that the black middle class has “lost its mind” if it believes Mr Cosby's argument downplaying the importance of race. Larry Elder, a conservative pundit, wrote a book about blacks who blame racism for nearly everything called: “Stupid Black Men”.

Mr Fryer eschews histrionics in favour of hard data. He is obsessed with education, which he calls “the civil-rights battleground of the 21st century”. Why do blacks lag behind whites in school? Mr Fryer is prepared to test even the most taboo proposition. Are blacks genetically predisposed to be less intelligent than whites? With a collaborator from the University of Chicago, Mr Fryer debunked this idea. Granted, blacks score worse than whites on intelligence tests. But Mr Fryer looked at data from new tests on very young children. At eight months to a year, he found almost no racial gap, and that gap disappeared entirely when he added controls for such things as low birth weight.

If the gap is absent in babies, this suggests it is caused by environmental factors, which can presumably be fixed. But first they must be identified. Do black children need better nutrition? More stimulation in the home? Better schools? Probably all these things matter, but how much? “I don't know,” says Mr Fryer. It is a phrase that, to his credit, he uses often.

Cool to be dumb

His most striking contribution to the debate so far has been to show that black students who study hard are accused of “acting white” and are ostracised by their peers. Teachers have known this for years, at least anecdotally. Mr Fryer found a way to measure it. He looked at a large sample of public-school children who were asked to name their friends. To correct for kids exaggerating their own popularity, he counted a friendship as real only if both parties named each other. He found that for white pupils, the higher their grades, the more popular they were. But blacks with good grades had fewer black friends than their mediocre peers. In other words, studiousness is stigmatised among black schoolchildren. It would be hard to imagine a more crippling cultural norm.

Mr Fryer has some novel ideas about fixing this state of affairs. New York's school system is letting him test a couple of them on its children. One is to give pupils cash incentives. If a nine-year-old completes an exam, he gets $5. For getting the answers right, he gets more money, up to about $250 a year. The notion of bribing children to study makes many parents queasy. Mr Fryer's response is: let's see if it works and drop it if it doesn't.

Another idea, being tested on a different group of children, is to hand out free mobile telephones. The phones do not work during school hours, and children can recharge them with call-minutes only by studying. (The phone companies were happy to help with this.) The phones give the children an incentive to study, and Mr Fryer a means to communicate with them. He talks of “re-branding” academic achievement to make it cool. He knows it will not be easy. He recalls hearing drug-pushers in the 1980s joking “Just say no!” as they handed over the goods, mocking Nancy Reagan's anti-drug slogan.

Blacks who do well in school are hungrily recruited by universities, which often admit them with lower test scores than are required of whites or Asians. The bar was first lowered for blacks out of a sense that America owes them a debt for past discrimination. Now universities are more likely to argue that racial diversity is valuable for its own sake.

But racial preferences are unpopular among whites, and the most blatant ones are, increasingly, illegal. The University of Michigan used to give applicants more points for being black than for getting a perfect score on the entrance exam. The Supreme Court deemed this unconstitutional in 2003, but ruled that less explicit preferences might be allowable.

When voters are asked if they want to ban racial preferences in the public sphere, they generally say yes. Since the 1990s, three states have passed referendums barring racial preferences, and four more may do so in November. Opponents of racial preferences argue that they are bad for blacks, too.

A study by Richard Sander of the University of California, Los Angeles, found that when the bar is lowered for black applicants to law school, they are admitted to institutions where they cannot cope. Many who drop out of top-tier colleges might have thrived at slightly less competitive ones. Mr Sander calculated that the net effect of pro-black preferences was actually to reduce the number of blacks who passed the bar exam. That is, racial preferences for black law students result in fewer black lawyers. John McWhorter, the author of “Winning the Race: Beyond the Crisis in Black America”, argues that lowering the bar for blacks also reduces their incentive to excel at school. “As long as black students have to do only so well, they will do only so well,” he says.

For every dollar that a white man earns, a black man makes only 70 cents. Such figures are sometimes bandied around to imply that nearly all of this gap is caused by discrimination. That is bunk. If a firm could really get the same work done 30% more cheaply simply by hiring blacks, someone would have noticed and made a fortune doing just that.

That said, blacks certainly face barriers in the job market. Two economists, Marianne Bertrand and Sendhil Mullainathan, sent out 5,000 replies to job advertisements in Boston and Chicago. Each fictitious applicant was randomly assigned either a black-sounding name, such as Jamal or Lakisha, or a white one, such as Emily or Greg. For every ten jobs the “whites” applied for, they were offered one interview. The “blacks” had to post 15 letters to elicit the same response. Clearly, some managers are racist. But many are not. And many firms are desperate to hire and promote blacks, if only to avoid lawsuits.

The short straw

Looked at more closely, the statistics are murky. White men are more likely to work than black men. The proportion of black men participating in the labour force fell from 74% in 1972 to 67% last year. Whites start more businesses, too. Only 5% of firms are black-owned, though blacks account for 13% of America's population.

A black woman with a degree earns as much as a white woman with a degree. But with a professional degree, the black woman earns 30% more (see chart 2). That does not prove that law firms discriminate in favour of black women—though they may. Another explanation is that a skilled white woman is more likely to have a rich husband (or indeed any husband), and so may have less incentive to maximise her earnings.

Even when blacks earn as much as whites, the whites are typically far wealthier. In 2000 the average white household in the bottom fifth of income-earners had net assets of $24,000. The figure for blacks was a piffling $57. Whites in the middle fifth were five times wealthier than their black counterparts.

Partly this is because whites inherit more. But it is also because of different approaches to investment. Blacks are more likely to put their money in the bank, notes Mr Fryer. Whites are more likely to invest in shares, which generate higher returns. Compound this over a couple of generations and the effect is colossal.

Another crucial factor is the collapse of the black family. The proportion of black babies born out of wedlock has nearly doubled since 1970, to 69%. And 70% of these births are to mothers who are truly alone, not cohabiting. Stable two-parent families accumulate wealth more easily than single-parent homes. Two salaries stretch further, two pairs of hands mean less need for paid child care. Two-parent families also find it easier to raise well-adjusted, studious children, who go on to start stable families of their own. Broken families, if middle class, find it harder to stay that way. And if they start in the ghetto, they find it harder to break out.

“Black life is not valued!” booms Michael Walrond, a popular pastor in Harlem. He is referring to the news that three police officers were acquitted of all charges after shooting dead an unarmed black man, Sean Bell, a few hours before his wedding. The cops fired 50 bullets, but the pastor says he is outraged by the figure of 31. Members of his mostly black flock know immediately what he means. Two of the officers were black and all of them thought Mr Bell had a gun. But it was the white officer who reloaded and fired 31 rounds. Mr Walrond's angry sermon draws cheers.

Afterwards, in his office, he agrees that it is not only whites who devalue black lives: black criminals do too. Mr Walrond, like many inner-city clerics, works hard to reform those who stray. But like Barack Obama's former pastor, Jeremiah Wright, he tends to assume the worst about his country. He finds Mr Wright's theory that the government concocted the AIDS virus to kill blacks “credible”.

He refers to the Tuskegee experiment between 1932 and 1972 when some doctors in Alabama deliberately neglected to treat black syphilis patients in order to study the disease's progression. That was an abomination. But it is hardly evidence that the government is bent on genocide.

From alienation to despair

Is the state racist? Those who think so often point to the criminal-justice system. A startling 11% of black males aged 20-34 are behind bars. Overall, black men are seven times more likely to be incarcerated than white men. Until recently, sentences for crack offenders (who are mostly black) were much harsher than those for powder-cocaine offenders (mostly white). Ex-convicts in several states are barred from voting, a penalty that deters no crime but signals to wrongdoers that they can never be full citizens again. “We are becoming a nation of jailers, and racist jailers at that,” reckons Glenn Loury, an economist.

Not so, says Heather Mac Donald of the Manhattan Institute, a conservative think-tank. Blacks are more likely to be jailed because they commit more crimes, she argues. In 2005 the black murder rate was seven times higher than that for whites and Latinos combined. Harsh crack laws account only for a smidgeon of the disparity in incarceration rates. In 2006 blacks were 37.5% of state prisoners; exclude drug offenders and that figure drops to 37%. And since black criminals' victims are mostly black, some argue that locking more of them up has saved many black lives.

In other ways, it is far from clear that the government is trying to keep blacks down. Affirmative-action policies mean that it provides jobs for a disproportionate number of them. It also allows blacks who own small businesses to charge 10% more than whites and still win federal contracts. “Small” is generously defined. A firm with 1,500 employees can qualify. Its black owner can be worth $750,000—excluding his home and business—and still be deemed “economically disadvantaged”.

Yet many blacks feel alienated in a way that is “vastly disproportionate to real-life stimulus,” frets Mr McWhorter. When New Orleans flooded, some speculated that the government had blown up the levees. Even cooler heads believed that the botched response stemmed from George Bush's indifference to black suffering.

Alienation has consequences. Amid the revolutionary fervour of the 1960s, says Mr McWhorter, many blacks learned that “America's racism rendered it unworthy of any self-regarding black person's embrace and that therefore blacks were exempt from mainstream standards of conduct.” The conventional wisdom about ghettos—best expressed in William Julius Wilson's book “When Work Disappears”—is that inner cities decayed because factories moved away. But the jobs often moved only a couple of bus rides away. Noting that millions of blacks moved halfway across the country to find work during the “great migration” in the early 20th century, Mr McWhorter wonders why so many of their descendants failed to follow suit.

He offers two explanations. First, a huge expansion of open-ended welfare in the 1960s enabled mothers to subsist without work. Until the mid-1990s, welfare often paid better than an entry-level job. Second, the counter-culture taught young blacks that working for “chump change” was beneath their dignity.

Bill Clinton fixed welfare and pushed millions of jobless women into work. Violent crime has also fallen sharply since the 1990s, despite the best efforts of gangster rappers to glorify it. Previously dysfunctional cities, such as New York and Washington, DC, are now soberly governed and better places to live in.

Yet many African-Americans are intensely gloomy. In a poll last year, only 44% said they expected life for blacks to get better, down from 57% in 1986. The subprime mortgage crisis, which will cost many black families their homes this year, will surely deepen the gloom.

Some blacks contend that racism has simply gone underground. Ellis Cose, a journalist, once wrote that even middle-class blacks suffer constant subtle racial slights, and that these are so distressing that they “are in the end most of what life is”. Other blacks think he exaggerates. Sometimes, says Mr McWhorter, the assistant trailing you in a store is just trying to sell you something.

If Barack Obama can only...

Taking the longer view, there is much to cheer. In every way that can be measured (a big caveat), racism has diminished in the past two generations. Inter-racial marriages are up sevenfold since 1970. Young Americans are far less likely to express racial animosity than their elders, suggesting that as old bigots die, they will not be replaced. And if Mr Obama becomes president, it would “raise the ceiling for everyone,” says Robert Franklin, the president of Morehouse, a black college in Atlanta.

“For me, racism is not going to be an obstacle,” says DeWayne Powell, a student at Morehouse. He recalls an incident when, en route to drop off his college application, he stopped to ask for directions. A white receptionist asked sneeringly whether he could read. “I laughed,” he says. “I thought: I'm on my way to fulfil my destiny, and you're stuck behind that glass.”