Home is at the heart of our most enduring and powerful tales: it's where Odysseus sails back to and it's why Dorothy follows the Yellow Brick Road. I found it at 28 Barbary Lane: a fictional address as resonant to many as 221b Baker Street, in a fantasy house that feels real, peopled by characters so beloved they're family – better than family.

Armistead Maupin began Tales of the City in 1976 – the year I was born. Told from multiple perspectives at a breathless pace it was first published as a newspaper serial, and featured real news: the Jonestown massacre, the rise (and fall) of the Moral Majority and even the Queen's visit to San Francisco in 1983. It was among the first fiction to tackle Aids. Nine novels later, Maupin has written The Days of Anna Madrigal – what he claims is the last instalment (he's said that before – please let him be fibbing again). Heart-warming and heart-breaking, Tales takes us from 1976 to 2012, from shared landlines to Facebook, from Quaaludes to Molly (MDMA), from the fringe to the mainstream. If you've somehow missed the multi-million-selling series turned mini-series turned musical by the Scissor Sisters let me tell you how very jealous of you I am.

You're about to meet the most fascinating people in the most fantastical place. Situated at the top of some wooden stairs on a fairytale lane in Russian Hill, 28 Barbary Lane is a boarding house ruled by the mysterious Mrs Madrigal. It's here that all the central characters live, love, grieve and from where they eventually leave. Mrs Madrigal calls her tenants her "children", her "logical family". To read the books is to join that family – doubly appealing for anyone escaping their own.



Reading on mobile? Watch Tales of the City Part 1/Chapter 1 here

It all begins with Mary Ann Singleton, the blond Cleveland refugee played to Panglossian perfection in the mini-series by Laura Linney. She is innocence personified in a pleated sailor dress, a midwestern secretary seduced by San Francisco. Let's just say she changes – a lot. The first character she meets is Michael "Mouse" Tolliver, a romance-addicted gay boy-next-door, and one of the truest gay characters ever written. Then there's the studly, liberal straight man Brian Hawkins, who quit law when he ran out of good causes and became a waiter while working the city's odds to his favour: a tameable fox. The ringleted far-out Mona Ramsey is on a quest to find herself when the answer is right beneath her coke-dusted nose. Finally, there is the kaftaned all-seeing mother – Mrs Madrigal. Her "Wedgwood blue eyes" keep watch on all comings and goings, yet she hides more than anyone. Seemingly incidental characters include the married-socialite-turned-lesbian-communist Ms Dede Halcyon-Day, the terrifyingly normal Norman Neal Williams and Jon Fielding, fiction's hottest speculum-slinger. And now, with The Days of Anna Madrigal, it's all coming to an end.

Or is it? Consider this your spoiler alert. Our old favourites, and some new ones too, decamp to the Burning Man festival in the Nevada Desert. Mouse, who dubs it "a Fellini carnival on Mars", has grown from twink to bear, as all gay men must. Now a settled sixtysomething, he's married to a much younger man called Ben and fears Burning Man won't have sofas, but goes anyway because he's Mouse. Even as a thirtysomething I identify with his move from all-night dancing to all-night box sets. Mrs Madrigal, now 92, warns: "Your regrets, my dear, are all about the things you didn't do." She's there, of course, getting more like Blithe Spirit's Madame Arcati by the minute. This orgy of self-discovery is the perfect place for yet more secrets to out. I sobbed for the last 30 pages but not, perhaps, for the reason you'd expect. My favourite scene is a very hot and very sweet threesome with Mouse, Ben and a beautiful youth dressed as a satyr. It is Mouse finally accepting, and getting off with, his younger self, but it is somehow not as narcissistic as it sounds.

Handing a newbie the keys to 28 Barbary Lane is one of life's simplest joys – like Mrs Madrigal taping a joint to Mary Ann's door on her first night. It's a way of reaching out, of saying "You're not alone" and also asking "Are you like me?"

"The house was on Barbary Lane, a narrow, wooded walk-way off Leavenworth between Union and Filbert. It was a well-weathered, three-story structure made of brown shingle. It made Mary Ann think of an old bear with bits of foliage caught in its fur. She liked it instantly."

Damian Barr at home in Glasgow

So did I. Shingle seemed impossibly exotic to a 14-year-old in a pebble-dashed council house in a village outside Glasgow. Then I was terrified of being myself, of being different, of being the only one. If this thing called Aids didn't get me then my Granny Mac would find out what I was up to with my best pal Mark and that would be that. I grew up not just gay but tall, speccy and scarecrow-skinny, the child of divorced parents from opposing sides of a sectarian divide. We all have our differences to bear, Maupin knows that. He celebrates them for us – especially when we can't.

I will never forget chancing on Tales in a Virgin Megastore in Brighton – Britain's answer to San Francisco. In life, as in the book, coincidences are never just that. Flicking through, I felt a flash of recognition that drove me, flushed with shame, to snatch it up lest it disappear. Over six million readers have done the same since the first title was published in 1978, making Maupin's series one of the bestselling ever. I've yet to meet the person who doesn't love Tales – though there are envious critics who dismiss them as froth and bigots who condemn them as filth. It would be like meeting the person who doesn't like chocolate.

Tales features all kinds of minorities and in the later titles gender-queer characters take up the reins of radicalism let go by Mouse and co as they win the big battles while growing older and comfier. But the books aren't minority interest. "Tales is inclusive," says novelist Patrick Gale, one of Maupin's close friends and author of a biography of him. "Other books were written in the ghetto for the ghetto and were all sex, sex, sex and if they aspired to be literary it was sex then suicide. Maupin's gay characters were among the first to have jobs and friends and lives."

Pity the poor gay fledgling that first walks into James Baldwin's Giovanni's Room. To avoid funny looks from the village librarian, I got on a bus to find a copy of Edmund White's A Boy's Own Story, which I'd heard about on something I shouldn't have been watching on BBC2 late one night. White's coming-of-age novel thrilled me – but it was dark. It didn't offer much hope. I occasionally stumbled across gay sex in the horror books I devoured but if the characters enjoyed it they were the first to get slashed/bitten/possessed. It wasn't looking good for me in fiction.

"Fiction is about trying on different identities," says Gale. "Tales invites you to be straight or gay or a bit of both, or even a 93-year-old transsexual. To live another life."

Long before the idea of a gay best friend became a cliche, Maupin explored the unique friendship between straight women and gay men. Mona's greatest fear is becoming a fag-hag. Maupin acknowledges the mutual ambiguities. Either could meet their Prince Charming and abandon the other but, even when he comes along, they choose to stay. In Mary Ann in Autumn, the penultimate book in the series, it's a much older, wiser Mouse that a divorced, moneyed Mary Ann turns to when she gets cancer. He is HIV positive but has been saved by the cocktail of drugs that came too late for so many. Together they survive. Heather, my high-school girlfriend, held my hand through my first gay breakup and gave a reading when I finally married my boyfriend Mike last summer. Together we survive.

In Further Tales of the City, published in 1982, Maupin maps amity between gay men and straight men – terra incognita still. In a typical moment, just when you think it'll all be ok, Michael and Brian are gay-bashed and Brian is stabbed three times. For all their sunny outlook the books are full of shadows. In hospital Michael cries: "They had even less reason to attack him than me. But he got the worst of it." His boyfriend Jon counters: "'Goddammit, Michael! How dare you talk like that? Brian doesn't think that. Mary Ann certainly doesn't. You're the biggest homophobe in the family. What the hell does gay have to do with anything? You taught me everything I know about being happy with myself.'"

And so I learned that I didn't deserve to be bullied either. Maybe men like Brian aren't always the enemy. Maybe I would find a friend, or even a boyfriend, to love and help me love myself.

"Mary Ann Singleton was 25 years old when she saw San Francisco for the first time. She came to the city alone for an eight-day vacation. On the fifth night, she drank three Irish coffees at the Buena Vista, realised that her Mood Ring was blue and decided to phone her mother in Cleveland."

That's how it all started when a new daily serial was launched in the San Francisco Chronicle. But Tales of the City did have an earlier incarnation. "The Serial", as it was first called, actually started two years before in the Pacific Sun, with an episode headlined "Social Hour At the Marina Safeway" in which a newly arrived Mary Ann meets a handsome local called Michael Huxtable. It ran for five whole weeks before the paper folded but Maupin was hooked, as were his readers. Two years later the Chronicle's editor was overheard at a cocktail party saying Maupin's serial was just the sort of "vulgar crap" needed to lure readers. The paper's society columnist rang Maupin, who braved the hard-drinking hard-faced editor and got hired. Other mooted titles included "San Francisco Stories", but Maupin chose Tales of the City saying: "It had a definite Dickensian ring to it." Indeed Christopher Isherwood exclaimed: "I love Maupin's books for very much the same qualities that make me love the novels of Dickens".

He wrote six weeks ahead in chunks of about 400 words, which is what makes Tales, like A Tale of Two Cities, so addictive. Maupin is a master of the cliffhanger – quite literally in the first book, when Mary Ann is left holding Norman's clip-on tie as waves crash below. Once you start, you can't stop.

The serial went "viral" long before the term existed. Friends faxed clippings to those outside the circulation area. In 1985 British Customs seized a Tales shipment destined for the Gay's the Word bookshop in London. I read the first books in secret and watched the mini-series with the volume turned down for fear of getting caught. Now I discuss the books with my mum. When Maupin tweeted me after reading my memoir, Maggie & Me: "Happy to have been there for you, kiddo!" I almost burst. Like Mouse, I couldn't imagine a happy ending: living in Britain's San Francisco with a cast of true friends, married to a man, in a society that no longer seems to hate me just for being myself.

'Nobody goes to San Francisco hoping to stay the same' … Marcus D’Amico and Laura Linney as Mouse and Mary Ann in the mini-series of Tales of the City. Photograph: c.PBS/Everett/REX

"Now they're beloved classics but to start they were hugely controversial," says Ken White, manager of Books, Inc in the Castro, still the gayest place on the planet. "I remember finding them shocking – I mean, nuns on roller-skates! I binge on every new title. We can't sell enough." Who's buying? "Mainly straight people because most gay people here have read them already – they're a rite of passage. Newbies send them to the folks back home as a kind of shorthand."

White previously managed A Different Light, a gay bookstore just along from Harvey Milk's camera shop. It closed in 2011: "In a sense, our goal was always to put ourselves out of business – once you've created a market, then inevitably it becomes mainstream. Gay content just isn't a barrier any more."

Not so in 1976. There are lots of coded references, such as Michael's insistence that his "friend" Robert, who Mary Ann tries to pick up, is "awful at hollandaise". I thought hollandaise was a sex act. Aged 14 I didn't yet speak the language of camp, but with Maupin's help I was learning fast. The dreaded diagnosis – "homosexual" – doesn't appear for well over a hundred pages. In the serial Michael didn't appear properly for six weeks. "His editor kept tabs with a wall chart," explains Gale. "There were two columns: one for heterosexuals and one for 'the others'. Armistead wanted to be entrenched before he took risks."

Maupin, who turned up for the job interview in a blue blazer with brass buttons, hadn't come out at work or to his blue-blood southern family. A virgin until 26, he was closeted all through college where he studied, for a while, law. In the navy Maupin toured Vietnam, which would have pleased his great grandfather, the Confederate general whose statue still keeps watch in the square of his hometown of Raleigh. Maupin's father led the entire family out of church when the minister preached for integration and Martin Luther King. Maupin Junior then went to work for the notorious racist Jesse Helms.

He sums it up best himself: "When I dragged guys home from Polk Street to my little place on Russian Hill some of them were quite horrified to see a photo of me shaking hands with Richard Nixon. But soon after I arrived here, I started to put my flimsy conservative politics against the kind hearts and generous spirits in San Francisco and began to change. Once I'd forgiven myself, everybody else got a pass." Maupin later denounced Helms from the steps of the North Carolina State Capitol on Pride Day saying: "I've changed and he hasn't."

Maupin finally came out to his parents and the world in the Chronicle, when Mouse wrote a letter to his mother, who was crusading with Anita Bryant against "the homosexuals" back in Florida. Thousands have since used the "Dear Mama" template. Including me.

"Being gay has taught me tolerance, compassion and humility. It has shown me limitless possibilities of living. It has given me people whose passion and kindness and sensitivity have provided a constant source of strength. It has brought me into the family of man, Mama, and I like it here. I like it."

More recently, Maupin has said: "My only regret about being gay is that I repressed it for so long. I surrendered my youth to the people I feared when I could have been out there loving someone. Don't make that mistake yourself. Life's too damn short."

It's this compassion and openness to change that makes Maupin and Tales so special. And it is San Francisco that makes it all possible. The city isn't just a setting – it's a character.

As it is also in the new HBO drama Looking, which follows a group of friends, mostly gay men, looking for better work, nicer apartments and a reason to delete Grindr. Maupin loves the city and he loves Hitchcock, whose films famously plumb its darker side. The Tales mini-series is scored with spooky music. Watch that first opening sequence and it's easy to mistake Mary Ann for Kim Novak's character in Vertigo. Maupin, like Hitchcock, is fond of a cameo and is featured writing at his desk, celebrating Mass in Grace Cathedral and leaving The Glory Holes.

A flock of green parrots honours us with a squawking fly-by on Telegraph Hill. "Right on cue," says Larry Rhodes, leader of the Tours of the Tales. "This city always gets it right."

Damian Barr (bottom right) on the Tour of the Tales.

I've finally come on a pilgrimage to 28 Barbary Lane, something I've wanted to do since I was 14. The end of the series seems a good time go back to the beginning. It's an unseasonably warm January day. There are 10 of us on today's tour, eight men and two women, of various sexualities. We've been walking up and down hills since meeting at the Buena Vista for Irish coffees at 9am. It's Larry's 60th and he has a walking stick but he's undeterred, running a couple of tours a year. We each have our own tale of the Tales.

"I'm so grateful to them," Larry says. "I knew I was gay when I was five but it took me 30 years to come to terms with it. One day a friend gave this book to me and that was that. Michael is kind of the gay everyman. He has the insecurities, the wishes, the victories. He gave me courage."

Rick Miller, Maupin's twinkly-eyed webmaster, is helping lead today's tour. He has the number 28 tattooed on his arm: "I identify with Mary Ann leaving the midwest and developing her 'logical' family. I feel her naivety and awkwardness and finally her confidence. I went to San Francisco shortly afterwards on the look-out for them all." As do thousands of others – the city recently awarded Maupin a Silver Cable Car award for tourism.

It turns out one of the women on our tour could actually be Mary Ann Singleton. "I grew up in the midwest," says Catherine Heenan, the cheerfully blonde anchor-woman for San Francisco's kron4 TV channel. "Moving here from Milwaukee, the first inkling I had of what a colourful, quirky, fun place it was came when a friend handed me Tales. I thought, 'If it's anything like this, I'm a happy woman'. That was 20 years ago. I've lured a lot of people here with those books."

But to where exactly? "That San Francisco has largely gone," says Don Romesburg, curator of the city's LGBT Museum and an associate professor at Sonoma State University. "For so many people Tales is San Francisco, it captures this golden era dense with nostalgia. But the tech booms mean Mary Ann couldn't now just arrive with no savings or job and find an apartment on Russian Hill." Back then Maupin rented his little red pentshack atop 1138½ Union Street for just $175 a month. Now Mrs Madrigal has sold up to stockbrokers who've made 28 Barbary Lane look like "a five star B&B". The city of free love has passed laws banning public nudity, which men get around with a carefully hung sock.

"The city has always marketed itself as 'wide-open'," says Romesburg. "Until the 1990s bus tours took tourists to see the exotic Orient in Chinatown, Beatniks in North Beach then a drag show at Finocchio's Club. The Tales books are a valuable historical source but they don't just reflect the city and wider culture, they've helped change them."

Mary Ann's mother's reaction in that very first chapter tells us all we need to know:

"'You can't just … run away from your family and friends to go live with a bunch of hippies and mass murderers!' … Her mother began to cry. 'You won't come back. I just know it.'

'Mom … please … I will. I promise.'

'But you won't be … the same!'

'No. I hope not.'"

Nobody goes to San Francisco hoping to stay the same. "People, like Mary Ann, still come here to find themselves, to make themselves," says Ken White. "That's what San Francisco is." "It's Armistead's version of the city but it feels real," says Gale. "I always think if I just go into the right bar or cafe I'll suddenly fall into this circle of wonderful friends."

The Tour of the Tales takes us all over Russian Hill and Telegraph Hill from Dede's penthouse to Beach Blanket Babylon, from real-life locations to places in our imagination. The most important stop on our pilgrimage is Macondray Lane – the inspiration for Barbary Lane. You'll find it between Leavenworth and Taylor streets. You'll know it when you get there because you've been there before.

I'm almost out of breath as I arrive at the foot of those wooden steps. Up and up they stretch, reddish new planks contrasting with faded silver ones. As I ascend they creak – just a little. From the first landing I glimpse Alcatraz. Up another flight and I'm in a leafy urban canyon where lacy tree ferns tower and ivy clambers. Up ahead I can hear old friends laughing. They're waiting for me. Just a few more steps and I'm there, I'm home.

As Mrs Madrigal says: "You didn't choose Barbary Lane. It chose you."

• The Days of Anna Madrigal is published by Doubleday. Damian Barr is the author of Maggie & Me (Bloomsbury, £14.99). He will be interviewing Armistead Maupin at his Literary Salon at the St Pancras Renaissance Hotel on 17 February. Amistead Maupin will be the guest at the Guardian Review Book Club at 6pm on 15 February at The Tabernacle, 35 Powis Square, London W11. Tickets: £12.