At 5.50am on 5 July 1967, a bill to legalise homosexuality limped through its final stages in the House of Commons. It was a battered old thing and, in many respects, shabby. It didn't come close to equalising the legal status of heterosexuals and homosexuals (that would take another 38 years). It didn't stop the arrests: between 1967 and 2003, 30,000 gay and bisexual men were convicted for behaviour that would not have been a crime had their partner been a woman. But it did transform the lives of men like Antony Grey, who had fought so hard for it, meaning that he and his lifelong partner no longer felt that every moment of every day they were at risk.

It is hard for us to imagine now how repressive was the atmosphere surrounding homosexuality in the 1950s. 'It was so little spoken about, you could be well into late adolescence before you even realised it was a crime,' says Allan Horsfall, who campaigned for legal change in the north west, where he lived with his partner, a headmaster. 'Some newspapers reported court cases but they talked of "gross indecency" because they couldn't bring themselves to mention it, so young people were lucky if they could work out what was going on.'

Antony Grey, who later became secretary of the Homosexual Law Reform Society (HLRS), describes having to make 'painstaking circular tours through the dictionary' to articulate the feelings he'd had since he was nine. The one thing he did manage to pick up was that 'there was a hideous aura of criminality and degeneracy and abnormality surrounding the matter'. Grey, a middle-class boy, fearful of breaking the law, remained 'solitary, frustrated and apprehensive' until he met his partner at the age of 32.

Grey is now 89 and has a civil partnership with that same man (Grey's partner has always remained anonymous and prefers to do so now). I met them at their house in north-west London, where we talked in a room overflowing with books. Grey is tall and used to look distinguished; he has had leukaemia and is gaunt now. But his memories of the period are precise. In the early days, they tell me, living together was a dangerous business. When a drunk coach driver crashed into their car outside their house in the night, 'the first thing we had to do was make up the spare bed. We knew from experience that if you called the police and they suspected you were homosexual, they would ignore the original crime and concentrate on the homosexuality.'

This was what happened to Alan Turing, the mathematician and Enigma codebreaker. In 1952, he reported a break-in and was subsequently convicted of gross indecency. Though he escaped prison, he was forced to undergo hormone therapy and lost his security clearance; he later committed suicide by eating an apple laced with cyanide.

For all that the law was draconian, it was also unenforceable. As a result, arrests often seemed to have an arbitrary, random quality. When Allan Horsfall became a Bolton councillor in 1958, he discovered that a public lavatory used for cottaging was well known to police and magistrates, yet there hadn't been a conviction in 30 years. On the other hand, there would be intermittent trawls through address books of suspected homosexuals, with the result that up to 20 men at a time would appear in the dock, accused of being a 'homosexual ring', even though many of them might never have met many of the others.

A case of this kind, involving eight men in Bolton, spurred Horsfall to set up the North Western Homosexual Law Reform Society (later the Campaign for Homosexual Equality). He explains: 'In this case, there was no public sex, no underage sex, no multiple sex. Yet they were all dragged to court and a 21-year-old considered to be the ringleader was sentenced to 21 months. I wrote a letter to the Bolton Evening News. They had four more letters in support and none against and the deputy editor was visited by the local police, who wanted to know if he thought this was what the people of Bolton really thought about the enforcement of this law.'

Horsfall thought it probably was and set up his campaigning group, which would play an important role in demonstrating to politicians that reform wasn't merely the preoccupation of a metropolitan coterie. When the objection was made, as it often was, that the powerful miners' groups wouldn't stand for legalisation, Horsfall was able to point out that he ran his campaign from a house in a mining village where he lived with another man and had never had any trouble with the neighbours.

In the mid-1950s, there was an atmosphere of a witch-hunt (probably not unrelated to what was happening in America with McCarthy), with consequent opportunities for blackmail. Leo Abse, who eventually piloted the Sexual Law Reform Act through Parliament, recalls that, as a lawyer in Cardiff, his fees from criminals suddenly all started coming from the account of one man. He investigated and found he was 'a poor vicar. The bastards were bleeding him. I sent for one of the criminals and told him if I had another cheque from this man, I'd get him sent down for 10 years. I sent for the vicar and told him to come to me if they approached him again.'

MPs on both sides of the House began to demand action. One or two newspapers ran leaders. And then there was another high-profile case in which the police were called on one matter and ended up prosecuting another. Edward Montagu, later Lord Beaulieu, contacted the police over a stolen camera and ended up in prison for a year for gross indecency. Two of his friends, Michael Pitt-Rivers and Peter Wildeblood, got 18 months. Their trial in 1954 probably played into the decision of the Home Secretary, David Maxwell-Fyfe, to establish the Wolfenden Committee to consider whether a change in the law was necessary.

As Lord Kilmuir, Maxwell-Fyfe led the opposition to law reform in the Lords, so it was ironic that he started the process. Perhaps he thought, by handing over to a committee, to shelve the issue. Perhaps he assumed Wolfenden would find against, in which case, he chose a curious chairman, because Wolfenden had a gay son, Jeremy. Antony Grey told me that when Wolfenden accepted the job, he wrote to Jeremy saying it would be better if he weren't seen around him too often in lipstick and make-up.

Allan Horsfall believes homosexuality was tacked on very late in the day to the business of a committee that had already been set up to look into the legal status of prostitution. (Certainly, its remit covered both; its findings were popularly referred to as the Vice report.) That would make sense of the choice of chairman, although it is also possible that, given the secretive atmosphere of the time, Maxwell-Fyfe didn't know Wolfenden had a son who wore make-up.

The Wolfenden Committee sat for three years and recommended that homosexual acts between consenting adults in private should no longer be illegal. Setting the tone for the discussion about law reform that would follow, it made no attempt to argue that homosexuality wasn't immoral, only that the law was impractical. The age of consent should, in the committee's view, be set at 21 (it was 16 for heterosexuals). The weedy reasoning behind this was that young men left the control of their parents for university or national service. In fact, it seems to have reflected a general prejudice that homosexuals were even more simple-minded than girls.

I met Leo Abse at his beautiful house overlooking the Thames at Kew where, he says, he is kept alive by his young wife Ania. He is 90 now and deaf, but mentally acute and still writing books. We talked in his first-floor drawing room as swans floated by outside. For all its shortcomings, the Wolfenden report is usually regarded as the key turning point in the fight for legalisation, the moment at which a government-appointed body said unequivocally that the law should change.

Abse insisted that its importance has been exaggerated. 'People talk sloppily about Wolfenden, which was not by any means a key turning-point. A myth has grown up: the myth of pre-Wolfenden and after. It was only a staging post. When I arrived in the Commons after Wolfenden, the vote against it was overwhelming. Ten years of struggle came after.'

It's true that an awful lot of lobbying remained to be done. The HLRS got off the ground in 1958, following a letter to the Times signed by 30 of the great and the good, including former Prime Minister Clement Attlee, philosophers AJ Ayer and Isaiah Berlin, poets C Day Lewis and Stephen Spender, playwright JB Priestley and various bishops. (From our perspective of the early 21st century, when the churches seem so afraid of homosexuality, it's interesting that in this period they consistently and visibly backed reform.)

Antony Grey became secretary in 1962, using the pen name he used for any letters he had published (his real name is Anthony Edward Gartside Wright): 'My father was dying. I didn't tell my parents I was gay until I was nearly 30 and they thought it was some foul disease. They were never comfortable with it.'

A long campaign ensued of talks to the WI and Rotary Clubs, university debates, public meetings and letter-writing. The meagre amount that the HLRS could afford to pay Grey was supplemented by means of a Saturday sub-editing job on The Observer, offered him by David Astor, then the paper's owner and editor, who was a supporter of reform.

The campaigning work was exhausting and often thankless and the opposition a mixture of vituperative and mad. Grey once caused consternation at a Rotary dinner when asked what homosexuals were really like, by answering, 'rather like a Rotary Club'. An opponent in a Cambridge University debate, Dame Peggy Shepherd, asked him over a nightcap at their hotel, 'Tell me, why are you so concerned about these unfortunate people?'

Various stabs were made at bringing the matter before Parliament, but the first really promising development came with a bill in the Lords in July 1965. It was sponsored by Lord Arran, an unlikely reformer: known to his friends as Boofy, he kept a pet badger. Grey recalls going for tea with him, with the creature in his lap. He had inherited the title because his older brother, who was gay, had committed suicide.

'He wasn't the sort of person you'd think would do it,' Grey says. 'But he was invaluable. He was related to everyone and was always saying things like, "I'll have a word with Cousin Salisbury about that." He was a bit mad - he referred to the bill as William - and he became an alcoholic while he was doing it. He more or less had to be dried out afterwards.'

For the opposition, Lord Kilmuir warned against licensing the 'buggers' clubs' which he claimed were operating behind innocent-looking doors all over London. But Arran, supported by the Archbishops of Canterbury and York, won his third reading by 96 votes to 31.

In the Sixties, the Lords led the way, quite unlike the situation in 2000, when the age of consent was finally equalised after the government invoked the rarely used Parliament Act to overrule a House of Lords that had thrown it out three times. Like the churches, the Lords has become more conservative about homosexuality over the years. The Catholic Archbishops of Westminster and Birmingham argued for exemptions in the 2007 Equality Act which would have allowed homosexuals to be turned away from soup kitchens and hospices.

Arran's bill ran out of parliamentary time, but its success meant the pressure was now on for the Commons. A Conservative MP, Humphrey Berkeley, tried to sponsor a bill in the lower house. He was gay and in many ways, the lobby, certainly Grey, would have preferred him. 'He was a nice person and not as quirky as Leo,' Grey says now. 'Both Arran and Abse thought that having got so far, they needed to make concessions, placate the implacable. It seemed to me that most people weren't worried about the details.'

Abse takes a different view. 'The House didn't like Humphrey Berkeley. He was gay and everyone knew. He was an enfant terrible who never grew up. I don't think he could have got it through.' Berkeley ran out of parliamentary time and then lost his seat at the 1966 general election. Back in the new Parliament, Abse gave notice in the July that he intended to move a 10-Minute Rule bill. By his own account, he was not the man the Home Secretary, Roy Jenkins (who wanted reform, fought for it in cabinet, guaranteed parliamentary time and assiduously sat through all the debates) would have chosen to pilot it through. 'We had a reconciliation before he died, but when Roy Jenkins talked to me in those days, he used to shut his eyes, as though he wanted to blot me out.'

Abse believes Jenkins would have preferred Michael Foot, for two reasons. First, he says, Jenkins wanted 'to bog down Michael', whom he saw as a potential rival in any leadership contest; and second, he 'thought I was too dangerous a character. I was too colourful'. He points to a shield on the wall, given to him by the Clothing Federation for being the best-dressed MP in Parliament. 'I used to dress up. My wife - my first wife - used to dress me up. By God, they needed some colour in Parliament! It wasn't only my narcissism. It was a part of opening up society. But I think Jenkins found it somewhat... he didn't feel comfortable.'

Abse's story is that he and Foot, who were and are great friends, outmanoeuvred the Home Secretary. Foot didn't want the job. 'He hadn't specially involved himself in the homo issue' (even allowing for the fact that he is 90, Abse's language seems a bit odd here), and when he realised what was involved, politely backed off. Jenkins did then give his support, Abse acknowledges, 'although on the way there were a couple of occasions when he lost his nerve and I didn't'.

Human-rights campaigner Peter Tatchell points out: 'The tone of the parliamentary debate alternated between vicious homophobia on one side and patronising, apologetic tolerance on the other.' The Earl of Dudley's contribution in the Lords sums up the level of the opposition's argument: 'I cannot stand homosexuals. They are the most disgusting people in the world. I loathe them. Prison is much too good a place for them.'

But, as Tatchell suggests, the tone of the supporters is, from this distance, hardly less cringe-inducing. No one mentioned equality or love. The consistent position was that homosexuals were pitiful and in need of Christian compassion. Abse argues now that much of this was tactical. 'The thrust of all the arguments we put to get it was, "Look, these people, these gays, poor gays, they can't have a wife, they can't have children, it's a terrible life. You are happy family men. You've got everything. Have some charity." Nobody knew better than I what bloody nonsense that was.

'My inspiration was ideology. It's a dirty word these days, but I was and am an ideologue. I am a Freudian. And I had been taught by Freud that men and women are bisexual. People should come to terms with their bisexuality, not repudiate it and become homophobic. You knew you were doing more than releasing thousands of people from criminality,' he explains. 'It was the start of opening up society to be more caring and sensitive. One was battling for all men and women to have a greater freedom.'

Commentators have argued over whether Abse was sufficiently ambitious with the substance of the bill, but there is no doubt that he was an adept tactician. He kept the mining MPs away from all the votes, 'calling in my debts'. He used his friendship with the chief whip, John Silkin, to ensure there was enough time and he drew the opposition's sting by gingering up a row over whether the law should apply to merchant seamen.

For a bit, it looked as though this arcane dispute might scupper the bill, but then Abse produced a compromise which, though patently absurd (merchant seamen could have homosexual sex with passengers and foreign seamen, but not each other) wrong-footed his opponents at a crucial moment. Right at the end, in the report stage, he managed to keep the required 100-plus supporters in the chamber all night so as to call for closure on the various amendments. On the last of these, he had 101 people there, which was all he needed but which, he says, 'shows how precarious the bill was, and it's why I get so damned annoyed when people say Wolfenden was a watershed. We got that bill through on one vote.'

Perhaps Abse is so anti-Wolfenden because his bill has been accused of being even more tentative than that report a decade earlier. Certainly, for those who had been lobbying, the act was a disappointment in several respects, not least in its confirmation of Wolfenden's age of consent. 'Of course, 21 was absurdly high,' Abse acknowledges now, 'but I wouldn't have had a hope of getting it through under that.' This does not seem to be his entire thinking though because he also says: 'Adolescence is a difficult time and many young men go through a homosexual phase. Great care is needed in that you don't corroborate them in their fixation.'

To make matters worse, the maximum penalty for any man over 21 committing acts of 'gross indecency' (which included masturbation and oral sex) with a 16- to 21-year-old was increased from two years to five years. Same-sex relations were also legal only in private, which was interpreted, as Tatchell says, as being 'behind locked doors and windows and with no other person present on the premises'.

While sex may have been legal, most of the things that might lead to it were still classified as 'procuring' and 'soliciting'. 'It remained unlawful for two consenting adult men to chat up each other in any non-private location,' Tatchell says. 'It was illegal for two men even to exchange phone numbers in a public place or to attempt to contact each other with a view to having sex.' Thus the 1967 law established the risible anomaly that to arrange to do something legal was itself illegal.

We shouldn't think this provision was quietly ignored either. In 1989, during the Conservative campaign for family values, more than 2,000 men were prosecuted for gross indecency, as many as during the 1950s and nearly three times the numbers in the mid-Sixties.

So, is Abse right that he got as much as he could in the circumstances? Antony Grey thinks certainly not; Allan Horsfall is more equivocal. It is hard to judge at this distance, although the experience of recent years suggests there is a lot to be said for moving swiftly to consolidate positions gained, as Stonewall has done in sweeping on from Section 28 to civil partnership to protections for sexual orientation legislation. Stonewall's chief executive, Ben Summerskill, acknowledges that in recent years, MPs with trade union backgrounds like John Prescott or Alan Johnson have been prepared to assert that equality means equality, which simply wasn't the case in the 1960s. Other Labour ministers of the recent past have been susceptible to arguments about their legacy, where Harold Wilson's government was mainly preoccupied with economic troubles and international crises.

Abse was disappointed in a different way by the aftermath of his Sexual Offences Act. 'Those of us putting the bill through thought that, by ending criminality, we'd get the gays to integrate. But I was disconcerted and frightened at first because they were coming out and turning themselves into a self-created ghetto.' Abse's views of integration sound rather more like wholesale capitulation to majority behaviour. But, in any case, he is wrong. Horsfall says: 'Nobody in the circles I moved in realised things had changed. It was 1970 before the Gay Liberation Front appeared and we were well into the Seventies before the Labour Party campaign for gay rights.'

Abse is disappointed that 'the gays' weren't more grateful. 'On my 90th birthday, I had lots of telegrams. I never had one word of thanks from any gay activist or lobby. When I've shown any reservations about the gays, they haven't forgotten. The ghetto suggests they are not at ease. They've got to have a gay world. Perhaps it was presumptuous to think they would integrate and become part of society. They use the excuse of external pressure and discrimination, but really it's not good enough.'

He is, unfortunately, taking a partial view. The single thing that more than any other has 'normalised' gay relationships has been civil partnerships. They could only have come about through lobbying, not by bien pensant intellectuals as before 1967, but by gay people themselves. And while it seems inconceivable now that we could ever go backwards, it is worth remembering the discrimination Abse dismisses was unchecked only recently.

Summerskill points out that recent events in Russia when gay activists, including Tatchell, were beaten up, possibly by plainclothes police, 'were not unthinkable in Britain 20 years ago'. Those archbishops arguing for the exclusion of homosexuals from hospices in 2007 offered a glimpse of a grimy homophobia that still sits mouldering on the underbelly of some British institutions.

The 1967 act was terribly flawed, but the world changed overnight for those like Antony Grey and Allan Horsfall who lived with their partners. The law also emboldened them and others to campaign for the right for those partnerships to have the same standing in law as any other marriage and for other rights to be themselves and have the same freedoms as everyone else. Mealy-mouthed, half-hearted, embarrassed by itself as it was, the act made possible the equality that has since been so painstakingly fought for.

A 40-year journey from shame to pride

1967 Sexual Offences Act in England and Wales decriminalises homosexual acts between two men over 21 years and in private. Scotland legalises it in 1980; Northern Ireland in 1982.

1971 First ever gay march in London, finishing with a rally in Trafalgar Square.

1976 Tom Robinson releases 'Glad to be Gay', which reaches No 18 in the singles chart.

1979 First ever gay television series Gay Life commissioned by LWT.

1982 UK's first HIV/Aids charity, the Terrence Higgins Trust launched.

1983 Labour Party candidate Peter Tatchell defeated in Bermondsey by-election after anti-gay campaign by tabloid press and local Liberals.

1984 Chris Smith, the UK's first openly gay MP, comes out while in office.

1988 Section 28, preventing the 'promotion' of homosexuality, introduced as part of the Local Government Act on 24 May.

1992 London hosts the first international Europride festival with a crowd of 100,000.

1994 Age of consent lowered to 18 from 21, despite unsuccessful campaign to lower it to 16, the consensual age for heterosexuals.

1998 Waheed Alli, one of the world's few openly gay Muslims, becomes the youngest and first openly gay life peer in Parliament.

2000 Government lifts ban on lesbian and gay men in the armed forces.

2001 Age of consent lowered to 16.

2003 Section 28 successfully repealed on 18 November.

2005 First civil partnerships take place.

Gordon Agar

· The following correction was printed in the Observer's For the record column, Sunday July 1 2007. The article above referred to Allan Horsfall as 'a former Bolton councillor'. Although in 1963 he was a resident of Bolton when he set up what became the Campaign for Homosexual Equality, he wasn't a councillor there. He was, however, a councillor in Nelson, Lancashire in the late 1950s.