You’ve never met Ta-Nehisi Coates’s father, but perhaps you know him generically. Paul Coates was that guy, the ageing black nationalist with his folding table full of dog-eared books. On sidewalks, at street fairs, at conferences, he coaxed wanderers over to inspect his wares – books like Black Egypt and Her Negro Pharaohs, Wonderful Ethiopians of the Ancient Cushite Empire, and, of course, the works of Marcus Garvey.

Seven years before the junior Coates published the wildly successful Between the World and Me, he wrote this memoir, brought out in January in the UK for the first time. He was, he tells us here, weaned on banal teenage fare – X-Factor comics and Narnia – and on books “brought back from the dead”, whose titles promised black people “militant action” and “a return to glory”. Coates’s father hustled out-of-print books by black thinkers shunned by the professorial establishment, not only on the streets but through a business run out of his basement, Black Classic Press.

Fatherhood and the word figure magically in this pithy, lyrical narrative. Paul Coates was “an intellectual, born as it happened among people who could not see a college campus as an outcome”. A Vietnam vet, he favoured tofu and Truffaut films. Black history was a forté of his; interpersonal skills were not. In one horrifying and yet comical scene, he visits his wife in the hospital when she gives birth. As she rests in her postpartum shine, he has no congratulations or tender words at the ready. Instead, he blurts, “Linda, I have another child on the way.”

Paul Coates had seven kids by four women. “He was called to fatherhood like a tainted preacher,” his son confides. “The root was his own alcoholic father, who seeded so many children that Dad simply lost count. He impregnated three sisters, and so Dad had aunts doubling as stepmothers.” Ta-Nehisi understood his father’s faults, but still regarded him as a prophet.

Ta-Nehisi Coates grew up in 1980s West Baltimore, almost a ghetto before it lay ravaged by Reaganomics, crack, and violence. A tender, ethereal, bookish child, he tried to keep up with his tough peers, groping for manhood in the dark. “My cheeks were fat,” he reveals. “I talked a lot, laughed in such a way that I gave the hardest kids around me permission to laugh. That same easiness made me soft, as I bounced awkwardly on my way to class in the morning.”

Coates writes so organically for black Americans, while artfully deflecting what Toni Morrison called 'the white gaze'

Throughout the memoir, his self-deprecation is charming and disarming. He writes about all the nothings he felt the planet pivoted on – girl crushes, driver’s licence tests, in-style kicks. Yet a momentous prescience overhangs. “On our life map,” he says, his father drew “a bright circle around 12 through 18. This was the abyss where, unguided, black boys were swallowed whole, only to re-emerge on corners and prison tiers. Dad was at war with this destiny.” In addressing such issues, this meandering yet powerful book throws down stakes both small and very large.

The Beautiful Struggle is polyphonic in the fascinating sense that Coates writes so organically for black Americans, at the same time artfully deflecting what Toni Morrison has called “the white gaze”. Coates resists portraying black people as they often appear in the white imagination and refuses to indulge prevailing white anxieties or expectations. Put crudely, the writing doesn’t inconvenience itself for the sake of white readers.

Recounting his disgust for the suburbanised pop culture of the 1980s, Coates writes: “Niggers were on MTV in lipstick and curls, extolling their exotic quadroons, big-upping Fred Astaire, and speaking like the rest of us didn’t exist. I’m talking S-curls and sequins, Lionel Richie dancing on the ceiling. I’m talking the corporate pop of Whitney, and Richard Pryor turning into a toy.” He steps out of his teenage skin, and with a hip-hop lilt subverts the mainstream.

He challenges conventional assumptions about black boys like him. After two assaults on his teacher and a bloody fight in a cafeteria catch up with him, he is expelled from high school. Coates describes his anguish at disappointing his father and himself, despite his best intentions. “No matter what the professional talkers tell you,” he writes, “I never met a black boy who wanted to fail.”

The Beautiful Struggle might take place in Baltimore, but it is not The Wire. Readers looking for black-dysfunction porn will be disappointed. The memoir is no cautionary tale pretending to document the loud, salacious bits of inner-city life, all the while glorifying or indicting them. There’s something quiet and elegant about Coates’s tribute to his complicated father, and to his boyhood city. Each small observation on his more-or-less mundane adolescence lures you into greater understanding; each comment on the ironies of race draws you further from apathy or political slumber.

Fatherhood is a vexed topic, particularly so for an author such as Coates. Roughly 50% of black American kids grow up in a household without a dad. A decades-long debate has centred on whether this mass of absentee fathers is to blame for “a culture of poverty”. Some of the most potent memoirs written by black men, The Autobiography of Malcolm X and Barack Obama’s Dreams from My Father, feature absent fathers, and (if only in this sense) don’t divert the white gaze. Though not as politically mesmerising, The Beautiful Struggle makes an enduring genre cliche – the father-son relationship – unexpected and new, as well as offering a vital insight into Coates’s coming of age as a man and thinker.

Having become a father himself, Coates wrote his celebrated, bestselling epistolary essay to his son. With a MacArthur fellowship, a National Book award and commanding robust speaking fees, he has become a fixture in the firmament of literary celebrity. The lovely irony is that he enjoys the renown his dad so keenly sought for his dusty library of neglected Afrocentrists.

• Rich Benjamin’s Searching for Whitopia: An Improbable Journey to the Heart of White America is published by Hachette Books. The Beautiful Struggle is published by Verso. To order a copy for £8.19 (RRP £9.99) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.