Christmas in my family is always a bittersweet affair, divided as it is between my dad, my mum, and my severely autistic brother's residential care facility. This year we were updating his photo album with pictures of him engaged in various college-mandated activities when my mother paused on one of him wielding a spade next to a massive pile of manure. He does not look especially happy, as anyone in that position, special needs or not, is not wont to look. It's a big pile of poo, after all. "That's my boy," said my mother wryly, "shovelling shit".

I thought immediately of this when I read Jamie Oliver's comments this week about his own special needs education, provided as it was by a teacher who is quoted in Oliver's biography as saying "to be honest, I never thought he would go far". Oliver is dyslexic and has only just read his first book now, in his late 30s. His view is that "fifty-odd per cent don't leave [school] with five GCSEs, A-C … in my mind we're half crap at education".

Research by the Prince's Trust, meanwhile, has revealed that young people with fewer than five GCSEs are four times more likely to turn to drink or drugs to deal with mental health problems. Those with few qualifications are twice as likely to be prescribed antidepressants, and a third of those who struggled at school say they regularly cannot afford to pay their bills. In short, it's looking as though society has failed a generation of children, many of whom will have been suffering from learning disabilities, diagnosed or undiagnosed.

My brother's education has been a battle, most of which has involved convincing the powers that be that he has potential at all. From the speech therapist who said, when he was three, that there "would be no point teaching him to speak because he'd only talk rubbish anyway", to the social workers who, eventually, we took to court to secure the care and education he deserved. Manure aside, he's in a brilliant place now, but for how much longer we do not know. It seems that, once the young person reaches adulthood, whatever their "mental age", the education system wanders off. We might need to gear up for another battle. It's a feeling with which many parents and families of children with special needs would be familiar; it can feel like a constant fight.

It was never a possibility that my brother would enter mainstream school, but I highlight his case because it illustrates how, to many, a learning disability seems like a life sentence – something that cannot be superseded ("I never thought he could go far") or improved on. At my school, children with learning disabilities were placed in "the unit" alongside those with behavioural problems on the verge of being expelled. They were mocked by the student body and, I'm depressed to say, some of the teachers.

Oliver spoke of the mockery he and his childhood friend and co-presenter Jimmy Doherty endured: their friends would sing "Special needs, special needs" to the tune of "Let it be, let it be". But that's not the worst a child can endure: "retard", "spaz", "spaccer", the nasty slurs that cast him or her out as "other", and led to me, perversely and probably wrongly, to feel grateful that my brother was profoundly rather than mildly disabled, because their words could never hurt him as they did me. On my way home, the kids would imitate the noises my brother made. Eventually, I moved school.

But this is what happens, when as a society you believe disability is something abnormal, a plight that belongs in a "unit" or makes you suitable only for certain menial tasks. I'm not saying that being swallowed up into a class of 30 is preferable, but complete segregation just makes things worse.

My late grandfather, who during the war had been evacuated to the same rural community where I grew up, would tell me about a man with Down's syndrome who used to ride the buses. They gave him his own bus driver's cap, and everybody loved him. I think it was supposed to make me feel better, but it made me feel worse. Decades later, local people showed affection for my brother, too, but I knew that many were scared of him. As a society obsessed with achievement, we still fear, mock and segregate those who fail to make the grade of "normality" and, while there are schools and professionals who do everything in their power for their charges, there are still more who would just hand them a shovel and have done with it.