In 1961, a beautiful model who graced the pages of Vogue appeared in the Sunday People under the headline: "Her" Secret is Out. April Ashley, then 25, was the first person in Britain to be outed as a transsexual, not long after she had travelled to Casablanca and survived difficult genital surgery. In subsequent decades, Ashley led the most extraordinary existence, getting up to mischief with aristocrats and actors as well as becoming an informal agony aunt for thousands of people struggling to understand their gender. Since her outing, however, she has never again worked as a model in Britain.

Ashley's exceptional experiences are typical of many trans people in Britain. "It was a very schizophrenic life," she says, referring not to switching gender but the combination of glamour and poverty, acclaim and abuse, she has encountered. Following Guardian columnist Suzanne Moore's spat with trans activists on Twitter, the vitriol directed at trans people by Julie Burchill in the Observer has caused many to wonder how much has changed.

Ashley, who is 78, penniless and last month collected her MBE from Prince Charles, is airily dismissive of Burchill, who called trans people "bed-wetters in bad wigs", among other insults. "I don't know where Miss Burchill goes to see people with crappy wigs on their heads. All the transsexuals I know are very smart looking and have good jobs," she says. "I do not wear a wig, by the way."

The transformation for trans people over the course of Ashley's life is astonishing. It is less surprising how little most people understand of trans lives. If gay activists traditionally asserted their right to be "different", most trans people have tried to "pass" for their new gender. There is no data on how many people are living as a different gender from their birth but activists estimate that 10,000 people in Britain have undertaken gender reassignment surgery, which was pioneered by German doctors on Lili Elbe, a Danish painter, in 1930. Elbe died from complications in 1931 and, although modern surgery is much safer, plenty quietly live their acquired gender without operations, particularly women "transitioning" to men, for whom genital surgery is more complicated.

April Ashley at Buckingham Palace with her MBE, December 2012. Photograph: Getty Images

It is eye-opening how trans people have only recently acquired the most basic of rights. Britain was one of the last countries in Europe to recognise a person's right to change their gender in law, and it was not until the Gender Recognition Act of 2004 that trans people could become a different gender in law. The court of appeal only established the right for people to access gender reassignment treatments on the NHS in 1999, and it only became illegal to sack someone who changed or planned to change their gender in 1997.

Perhaps it is inevitable that the small group of trans people, who have had legal recognition for a mere eight years, say they still suffer discrimination, prejudice and violence. A study of 2,600 trans people in the EU in 2008 found 79% suffered transphobic abuse – verbal, physical or sexual violence – in public. More recently, Transgender Europe logged 265 reports of murdered trans people in the 12 months to November 2012; 126 were in Brazil, with one in Britain (although not all these deaths are proven to be the result of transphobia).

Paris Lees, 25, an eloquent, media-savvy campaigner and editor of digital trans magazine META, was violently assaulted and abused in the early days of her "transition" when she did not "pass" so well for a woman. She knows countless people who are harassed in their homes and called "freaks" or "perverts" on the streets every day. "Try putting on some lipstick and holding a handbag and going out there," she says. "There are two types of trans people: trans people who are lucky enough to 'pass' – their lives are pretty much like yours – and people who are identifiable as trans. Their lives are living hell. They cannot go out of the house without being abused. There's a long way to go and it has to change. People need to feel safe walking down the street."

Most trans people barely notice everyday harassment. Stephen Whittle, professor of equalities law at Manchester Metropolitan University, still gets stones thrown at the house where he has lived for 20 years with his wife, Sarah, and their four children. He has also been abused in the lecture theatre by students, who have called it an "abomination" that he has children. "On the whole you can get by in life without too much hassle, which is pretty different to 20 years ago when every moment of life was hassle," he says.

Whittle, who "transitioned" nearly 40 years ago, was one of three trans men and three trans women who did an unusual thing in 1992: they went to meet Liberal Democrat MP Alex Carlile in Westminster. The unusual element was not the meeting but the fact that they travelled together – at the time, trans people never dared to because it increased the likelihood that they would be spotted and abused. These six wanted to start a campaign group; Carlile advised them to avoid the word "transsexual". So, in Grandma Lee's teashop opposite Big Ben, an anodyne name, Press for Change, was chosen.

Stephen and Sarah Whittle with their family in 1999. Photograph: Don McPhee for the Guardian

The 80s, remembers Whittle, had been "dreadful years". As soon as his trans status was discovered he would be sacked; it was the same for every trans person. At the job centre, the adviser would call out, "Miss Stephen Whittle". At his teacher-training medical, the doctor told him they couldn't have "his sort" in teaching. "It was very, very hard, not just on us but on the people we fell in love with and lived with. We felt like we could never, ever win this battle. All these years on, we have sort of won the battle."

For decades, Ashley's life itself was a source of some of these battles, as one of the few widely known transsexuals in Britain alongside Jan Morris, who completed her transition in 1974. The annulment of Ashley's marriage to the Hon Arthur Corbett (in court he was judged "deviant"; she "a man") in 1970 was a humiliation for Ashley and a great setback for trans people because it was established that a person must remain their birth gender in law. Before that, trans people were furtively altering their birth certificates, or passports, and accessing medical treatment.

Christine Burns is one of a generation who vividly remembers reading about Ashley in the papers when she was a young child. (Ashley appeared in a six-week special in the News of the World: "They were one of the very few who paid me and they behaved impeccably. I was very sad when the News of the World closed," says Ashley.) The existence of someone like her in the public eye was a great comfort for Burns. In the 90s, when she was chair of the Women's Supper Club of the local Conservative party association in Cheshire, she quietly joined Press for Change. Even then, the new activists dared not be openly trans. "The thing that held us back in the 1990s campaigning was that fear of being out," admits Burns. Eventually, she came out in 1995; she jokes that she realised she was more embarrassed to be a member of the Conservative party than openly transsexual.

The writer and historian Jan Morris completed her transition in 1974. Photograph: Colin McPherson/Corbis

Much of their campaigning remained on the quiet. The passage of the 2004 law to give trans people legal status was "remarkable," says Burns, because "the government was able to pass an entire act in parliament without anyone throwing a fit in the press". In popular culture, the activists became more forthcoming in their attempts to increase popular understanding of trans issues. Although the arrival of trans character Hayley Cropper in Coronation Street in 1998 was one breakthrough, Julie Hesmondhalgh, who plays Cropper, is a non-transsexual woman. Some believe one sign that minority groups are not taken seriously is when characters in popular culture are not played by members of that group (from the Black and White Minstrels of the 60s and 70s to non-disabled actors taking disabled parts). "I can advise any casting directors that there are plenty of transsexual actresses," says a medical professional involved in transition treatments. More inspiring for many younger trans people was the victory of Nadia Almada in Big Brother in 2004. Equally significant for a less visible part of the trans community was trans man Luke Anderson's Big Brother victory last year. But there still persist the likes of Little Britain and hundreds of other belittling jibes about "trannies" and "chicks with dicks".

Most trans people I speak to say the biggest issues they face are not media stereotypes but legal rights and access to healthcare. When trans people were allowed to legally register their changed sex in 2005 there was an awful tangle over marriage. Fearful of creating a situation where two women could be legally married, the government decided that trans women who married when they were still men must have their marriage annulled to receive legal recognition as women. So while Ashley finally became a woman in law (with a bit of help from John Prescott, with whom she worked with in a hotel in the 1950s – "a very charming young man", she says), married couples who have stayed together through one person's transition still have to divorce if the trans person's gender is to be legally recognised. "Imagine the wife of someone who transitions from male to female; I cannot think of an issue that challenges your marriage more," says Burns. "So this law is absolutely indefensible. It's a real slap in the face to the partner. The law considers relationships for trans people and those who love trans people to be disposable."

Sarah Brown, an openly trans Lib Dem councillor in Cambridge, was a man when she married Sylvia in 2001. They stayed together through Brown's transition, which began in 2005. Brown wanted to be recognised as a woman in law and she and Sylvia convinced themselves that annulling their marriage and becoming civil partners would "just be a bureaucratic exercise that didn't mean anything". When they got their decree of nullity, however, "We realised we had been wrong. We left the court holding hands, in tears," says Brown. Far from being healing, the words of their civil partnership a few weeks later were "a kick in the teeth". Even their marriage certificate was confiscated. The proposed marriage equality legislation will not allow people such as Brown to have her original marriage recognised again.

This is a source of personal heartache, but Brown is convinced that trans people's biggest single problem is access to decent healthcare. Within 24 hours of her creating the hashtag #transdocfail, she had been inundated with 2,000 tweets of trans people's negative experiences at the hands of medical professionals. "It revealed a massive level of abuse. If it was happening to any other minority it would be on the national news," says Brown. Another ongoing battle for trans people is to revise the World Health Organisation's International Classification of Diseases (ICD) so that transsexuality is not listed as a mental illness. (Ironically, this was added to the ICD around the time homosexuality was removed; campaigners such as Burns say it is useful to be in the ICD to help trans people access healthcare but it should be "less stigmatising".)

When they finally gain access to transition procedures, most trans people are positive about their hormone treatment or surgery. More of their complaints, however, concern everyday health problems and ordinary care from GPs. Many trans people feel "absolute terror" when faced with revealing their medical history for an orthodox operation or treatment, says Brown, and believe doctors then treat them differently. "It's not just the NHS that is institutionally transphobic, it's the whole medical establishment," says Brown. "This attitude that you're not treating the person, you're treating a condition. The moment people realise you're trans, it seems they can't see anything else."

Paris Lees, editor of META, is a campaigner for trans people. Photograph: Rachel Saunders Photography

James Barrett is the lead clinician at the Charing Cross Gender Identity Clinic, the largest and oldest in the world, which receives 1,400 NHS patients a year, a figure that is doubling every five years. About one in five referrals end up having genital surgery. The media is obsessed with stories of regret; in fact, post-operative trans people wanting to return to their original sex are "vanishingly rare", says Barrett. Of the 6,000 or so NHS patients he has seen over 25 years, just two have permanently reverted to their original gender role. Barrett understands some trans people's frustration with the glacial pace of gaining access to surgery but says if it was made easier – or the selection processes less stringent – there might be proportionally many more regrets.

Barrett admits his work is not well regarded among many health professionals and is critical of some GPs who treat trans people. "It's not a majority, but it's an extremely substantial minority" he says of GPs who are reluctant to refer patients on to identity clinics, or are unwilling to prescribe the hormones they will need for the rest of their life when guided by consultants at a specialist NHS clinic. "We have persistent problems with GPs who won't prescribe for patients even though to do so is safe," he says. (A Dutch study found mortality rates among treated trans people no higher than anyone else; Barrett's clinic's oldest former patient is 92.) In fact, says Barrett, it seems that some GPs are prejudiced or ill-informed: one stated it was against her Christian beliefs to prescribe hormones; another recently insisted no such treatment was available on the NHS.

"This is a group who are somehow not taken seriously. They are a bit like Gypsies, whom it still seems OK to make racist jokes about," says Barrett. "It seems to be thought comedic to make cruel jokes about them. Nobody would make Dick Emery-type jokes about gay people now. This isn't a pantomime. These are real people with real lives."

Barrett understands the potential controversies inherent in trans people asking for medical help paid by the taxpayer but insists that people who have gender reassignment surgery subsequently contribute far more to society than they ask of the NHS. For example, they earn more after surgery. "If you work out how much more they pay in taxes, they fund their own treatment and then some. If you don't treat people, they tend to be really miserable and off work and on sickness benefits and in hospital and then they cost the taxpayer, when they could be net contributors in financial and social terms."

Part of Burchill's critique of trans campaigners was to suggest they are a small, educated minority who punch well above their weight. Whittle admits trans people tend to be well educated but says this is a legacy of them having no jobs to go to. Whittle, Lees and Burns all came from humble beginnings and are now smart, networked individuals.

In 25 years, Barrett has seen trans people become "a networked bunch" – more so than other people, he thinks – thanks to the internet. Lees, who also works for Trans Media Action, says social media is the "essential catalyst" for the transformation of trans people in society. "Society is in transition and we've woken up from the operation and there's no going back. We can't pretend that trans people don't exist any more," she says. "People have been taking the piss out of trans people for 60 years. The narrative on trans issues has been controlled by people who have no understanding of them. Social media is about us grabbing the narrative back and telling our own stories – this is our reality, this is what we go through and this is what matters to us. We're here, we're in your face, we definitely exist. That's the most important thing – realising we exist."

Later this year, an exhibition about April Ashley's life – from electric shock treatment in a mental hospital to being admired by Albert Einstein, pursued by Dalí and apparently sleeping with Omar Sharif and Peter O'Toole on the same holiday – will open at the Museum of Liverpool. Ashley doubts whether trans people will ever be completely accepted in society, but she believes life is improving. "There will always be a great deal of prejudice but I do think things are getting a lot better, and trans people are also getting a lot better," she smiles. "Transsexuals are famous for their sense of humour, their gentleness, their kindness and being nice people because they've been through an awful lot. It's a terrible ordeal to go through to have a sex change. People don't do it willy-nilly. It makes you humble."