I am sitting in a cafe in my home town of Margate, with a man whom most residents, myself included, were once afraid of. Ricardo Scott, all tattoos and shaven head, is wearing a T-shirt that says simply: SKINHEAD.

For the last few weeks, as part of a commission from the town's new Turner Contemporary gallery, I have been moving a shiny silver Airstream caravan along the town's seafront, between its housing estates and car parks, like a giant Monopoly piece, collecting memories and photographs of Margate's youth subcultures. The town has figured strongly in the history of teddy boys, mods, rockers, punks, skinheads, rockabillies, rude boys and soul girls – but few have recorded the voices, views and looks of this largely working-class well of creativity. What has been recorded consists largely of photographs from a 1964 Whitsun bust-up between mods and rockers, much of which was transposed to Brighton for the 1979 film Quadrophenia.

"The best sight [in those days] was a load of skinheads walking down the road," says Scott, now a father and roadworker. "At that age, you would start to put fear into people and enjoy it when people got out the way. There was a lot of violence, which was there whether you wanted it or not. But that was the time, wasn't it? When you look back, you think it was stupid, but you are older now."

Scott and I stroll to Marine Gardens, a regular hangout in his youth. I photograph him at an angle that has him dwarfing a local tower block. I realise that I am often casting my subjects as heroes, shooting low and looking up – an unwitting artistic tic. Similarly, I shoot a biker-turned-lifeboat crew member seemingly nine feet tall against the depth measurements on his boat.

These new portraits now hang in the Turner Contemporary, next to portraits of their younger selves; the instinctive creativity of youth, the desire to express themselves, is evidenced in their home-cut hair and outsider uniforms. Scott's younger self is pictured against a backdrop of newspaper cuttings and headlines torn from sandwich-boards bolstering the notoriety of local skinheads.

By the early 1980s, Margate was tipped as a potential target for riots. The town had unemployment levels that compared with the north, and there was unease on the streets. Groups of more than two youths were separated by police; skinheads visiting for bank holidays were held in pens at the police station. I was a skinhead and then a punk during this period, informed by my school and anyone else who cared to chip in that there was no future for me. We were the "feral youth" of our day.

Finding all these former folk-devils who had turned into lovely adults, with a great enthusiasm for family and for Margate, was a constant surprise. There was the larger-than-life former rocker whose Hell's Scorpions gang used to meet in the basement of the cafe where I talked to Scott; the rockabilly who became a social-worker; and the girl who ducked out of the Margate Skins when the violence became too much. "I remember running into Woolworths and hiding from some mods in a photo booth," says Tracey Butterfill, who has moved back to the area to retire after working in the City. "I did see a lot of fights. I remember taking one of the guys to hospital once, as he had been stabbed."

Standing Butterfill against the seafront clock tower, I created a portrait that felt like a cross between Jean Seberg and a Soviet Union-era statue. The photograph of her as a young woman that now hangs in Turner Contemporary could be a still from Shane Meadows' This Is England; the juxtaposition of the two portraits says as much about what she has retained from her youth as it does about what she has rejected.

Margate's old networks are mostly intact; where they were not, I found myself becoming a human version of Friends Reunited. I encountered a former skinhead friend painting Turneresque seascapes and selling them next to the Turner Contemporary; I also heard stories about people whose lives had been taken by drink, drugs and despair. People opened up. The idea that their portraits would hang in the new gallery, a place that several still saw as "not for the likes of us", made the project doubly exciting.

Former rocker Ian Blyth had followed an unorthodox path, becoming a collier and then a police officer. He retired five years ago, leaving him plenty of time to ride his Triumph Bonneville, wearing his leathers and his 59 Club badge. Over not-so-frothy coffee, Blyth reminds me that being a member of a subculture is something that never leaves you. "You don't live in the past, your past lives in you," he says, dressed in an outfit similar to one he would have worn in 1964, complete with white silk scarf.

Blyth adds that your past can stay with you for longer than you might wish, as it did for those arrested and fined or imprisoned for fighting on the beaches. "Everyone liked to bask in a bit of notoriety," he says. "After the headlines about a bank holiday meet on the seafront between the mods and rockers, you would strut around going, 'I'm a rocker.' Young people are always going to rebel, whatever culture they choose. It was a good feeling, until you get sensible and think, 'That was a bit daft.'"

The parallels between my subjects and those who took part in the unrest across England this summer are not hard to see. Ignored youth with little hope will find a way of getting noticed, whether through fist fights and outrageous haircuts, or by covering their faces and looting JD Sports. I never sat down for coffee with an old skinhead to become wiser about the world at large – but history, however recent, has a funny way of repeating itself.