On Tuesday 24 August, a storm was brewing off the North Wales coast. As gusts reached 50mph in the village of Valley, on Anglesey, two sodden police officers stood at a front door on a close of modern bungalows above the beach. They knocked and knocked. Eventually someone ran out into the rain to tell them that Ellen and Ian Williams were away on holiday in the US. While detectives tried to locate the Williams abroad, someone found Ellen's father, John Hughes, who lived around the corner and broke the news: his 31-year-old grandson Gareth, a boy who had always been fit and healthy, was dead. Grandpa John was so distraught that he tripped, cut his head and had to be taken to hospital.

Then Gareth's uncle, William Hughes, who lived on a farm in nearby Trefor village, got the call. "Gareth couldn't be dead? Surely not?" Uncle William sat numbly with Gareth's 86-year-old great aunt before their open fire, staring at the porcelain figures on the mantelpiece. Gareth had been an exceptional boy, selected to work as some kind of analyst for GCHQ, the government's secret listening station in Cheltenham, but the family never talked about it. And neither did Gareth. He'd been home just three weeks earlier, recalled Hughes, cycling the lanes on his race bike before the whole family had got together. But his life was "now being lived a long way from the family in Anglesey," Hughes said, "so some things we couldn't know."

On 25 August, Ian and Ellen Williams touched down at Manchester airport and were met by police, who officially broke the news that two days earlier in London, a man believed to be their son had been found dead. The body had been found padlocked inside a sports bag in the flat where Gareth was staying. It could have been there for up to a fortnight. As they were escorted to the capital, to join their daughter Ceri, there were news reports everywhere: "British spy murdered." Apparently he had been "stabbed to death". According to stories, Gareth was a brain-box, a techie who "worked with codes", as far as they knew. Was he also a spy, they wondered?

For Ellen and Ian, Gareth's death was a horror story that "destroyed" them. For the rest of the world, it was a rare glimpse into the clandestine world of espionage. His death was compared to that of Georgi Markov, the Bulgarian dissident killed by a poison-tipped umbrella in London 32 years before, and the 2006 murder of Russian dissident Alexander Litvinenko. Cold war details increased public fervour. The building in which the body was found was owned by a private company, registered offshore, called New Rodina. Rodina was Russian for "motherland". Did real spies read John Le Carré too?

In the midst of lurid speculation, Ian and Ellen Williams reached London. In normal circumstances, they would be required to make a physical identification. However, given the degree of decomposition, police would rely on family photos. Was Gareth murdered, his distraught family asked. Despite launching a major investigation – Operation Finlayson – Scotland Yard were finding it a hard question to answer. One major problem facing investigators was the length of time that the body had lain undiscovered. Another was that they were being blocked by the security services from probing too deeply into Williams's life. Detectives had no idea what Gareth Williams really did or who he was, and had been unable as yet to find anyone close to him. They needed the family's help.

Gareth's sister Ceri, a physiotherapist, and her husband Dr Chris Subbe, a consultant at Wrexham Maelor Hospital, told police they had seen Gareth in June, when he had taken them for tea at the Ritz in London to celebrate their second wedding anniversary. He'd been in the capital for more than six months, but couldn't say what he was doing.

The three spent that afternoon "laughing and joking", and afterwards continued to talk frequently by phone. Ceri last spoke to her brother on 11 August, after he returned from a short trip to the US. He now intended to "tidy things up", before returning to GCHQ in Cheltenham on 3 September.

Ceri's dates matched what police learned from Gareth's former landlady, Jenny Elliot, who had rented him a room in her house near Cheltenham for the 10 years he had worked for GCHQ, before transferring to London last year. Gareth had called her in August, asking if he could return. "Like we were going to say no," Elliot says. "He was about the best lodger you could ever have. I mean, you never heard a word from him. He had no TV, record player, or anything."

Ceri had called her brother again about a week later. No answer. She tried repeatedly, becoming worried. Gareth was like a Swiss watch. Now she contacted the police; the security services were already concerned, as Gareth had failed to show for work. On 23 August, at 6.30pm, a uniformed officer was sent to Williams's top-floor flat in a Georgian townhouse in Alderney Street, Pimlico. It was only a few hundred yards from the MI6 headquarters at Vauxhall, and was used by the Secret Intelligence Service as a safe house. A lettings agent who held a spare key showed the policeman in. He was asked to wait downstairs as the officer went up and entered the curtained rooms. The place was "spotless": two iPhones, some sim cards and an Apple notebook sat on a table. Then he entered the bathroom, and found a holdall in the bathtub. Red liquid seeped from it. Inside was a body, in such a contorted position that he thought its "legs and arms had been cut off". Radioing for assistance, the policeman noted that there were no signs a struggle. This was not a robbery or home invasion. It was a "neat job", a euphemism for a professional kill.

Given his clearances and access to classified material, Williams's death triggered alarm across Whitehall. MI5 agents swept through the Alderney Street flat, followed by detectives from the Homicide and Serious Crime Command, assisted by SO15, Scotland Yard's Counter Terrorism Command. Alderney Street was cordoned off as Home Office scientists began processing the scene. Once the spooks had departed, it was down to the murder squad to study the evidence. So far, they had a body in an "advanced state of decay". There was no weapon and no sign of forced entry or a struggle.

Concentrating on the holdall and its contents, detectives established that Williams had not been stabbed, shot or hacked to pieces. On August 25, Home Office pathologist Dr Ben Swift carried out a post-mortem that, together with the first batch of toxicological tests, came back "inconclusive". "If we don't know what to look for, and are not guided by what we see or smell on the body, we cannot find it," says a forensic scientist who worked on the Litvinenko case. "We do 50 obvious poisons. Fifty rare. We can do the isotopes. Litvinenko alerted us to that. But in the absence of a specific direction, the possibilities are as limitless as a killer's imagination… and we cannot test for that."

Detectives set out their theories. One reading of the crime scene pointed to a forensically aware hit man. Slender 5ft 7in Gareth had been stuffed inside the near-airtight red North Face sports bag and placed in the bath, containing any spillage and minimising odours, too. The heating was on, increasing the rate of decomposition, which significantly lessened the chances of retrieving evidence from the corpse.

Or perhaps the body in the bag was evidence of a stage-managed "personal event", masterminded by a controlling individual. Was this a suicide (with Williams acting on his own)? Or a sadistic or masochistic sexual act gone wrong (with Williams engaging in some kind of auto-erotic asphyxiation?). "If you can imagine it, then we are investigating it," a detective said in late August. But what would shape police inquiries was the dawning realisation that Gareth Williams had not been alone.

Returning to the holdall, studying the zips and lock, police became certain that he could not have locked himself inside. And there was further proof that someone else must have been in the flat with him: his front door had been locked from the outside.

Criminal psychologists studied the case. One, who spoke anonymously, thought his death might have been a masochistic ritual that had gone wrong: "For a retentive individual with an ever-green mind like Gareth Williams, always on duty and in control, the bag could have been a furtive release."

The Williams family was adamant that the Gareth they knew would not have behaved this way. And, for now, the family was all the police had, since they had found no close friends or lovers. Another pressing problem with their theories was that detectives had failed to turn up any other suspects or accomplices.

Police, pathologists and forensic investigators agreed on one thing only. They would have to focus on the 12 days between the last call Gareth made to Ceri and the discovery of his body on 23 August. Pulling mobile phone records and credit card transactions, scouring hours of CCTV footage, the Operation Finlayson team dug deep, as ever more lurid stories circulated.

"Murdered MI6 worker Gareth Williams was a secret transvestite who may have been killed by a gay lover, detectives said yesterday… Cops found women's clothing that would fit him at his Pimlico flat in central London." The Sun ran with this version of events, while other newspapers, quoting similar unnamed sources, reported that cocaine had been found, a cache of gay pornography, a small armoury of S&M paraphernalia. Williams was also said to have frequently paid for male escorts.

Senior detectives angrily rebutted the stories. Uncle William in Anglesey thought they read like a concerted smear, and wondered why anyone would want to destroy Gareth's reputation after he was dead.

Like most well-constructed character assassinations, however, this one was founded on a grain of fact that made it that much harder to quash. Detectives had been quietly investigating whether Gareth Williams was gay. They had canvassed witnesses in the Vauxhall Cross village of gay clubs and bars, a short walk from Alderney Street. They had also studied Williams's computers and reading matter, his journals, magazines and computer cache. Despite outright denials from his family, detectives were privately certain that Gareth was gay, although they had been unable to find trace of any sexual encounters. They had also come to believe that his sexual orientation was not central to the inquiry (although his sexual preferences might be). There was no evidence of Williams paying for escorts, buying S&M equipment, using porn or drugs of any kind. "This man didn't really even drink," one frustrated detective said.

The police felt they were being hurried along by other parties, keen for the scandal to go away. "Someone, somewhere, who has access to case material, is saying, 'He's queer and asked for it', rather than waiting for the outcome of the case," one veteran detective said.

In North Wales, the family lit their fires, closed their curtains and privately pondered if anyone had ever understood the real Gareth.

At Ysgol Gynradd Morswyn, Gareth's primary school, a short hop between Valley and Holyhead, head teacher Islwyn Williams remembers him as an exceptional pupil. At five, Gareth made a beeline for the school computer and never left it. While most of his classmates were still playing with Lego, Gareth had a GCSE in maths. At the age of 10, he went to secondary school, gaining A grades in A-level maths and computer science at 13. The school contacted Bangor University, which accepted him as a part-time student on a maths degree course. "He was the best logician… with the fastest brain I have ever met," said Geraint Williams, his maths teacher.

Fellow pupils remember Gareth as a loner. Dylan Parry, 34, thought him "isolated" by his intellect. His abilities shut him off from everyone else. He was naive. Parry believed someone "could easily take advantage of him". He recalled Gareth travelling by train each week to university, while still living at home with Ian and Ellen. Too clever for school. Too young for university.

The family downplay his isolation. They point to their closeness and his sporting prowess. By the time he was 17, Gareth had followed his father into competitive cycling. Keith Thompson, of Holyhead Cycling Club, with whom Ian and Gareth rode, says: "He was a really lovely young man, but wasn't a great conversationalist. Gareth wasn't the sort to go to the pub after a race."

In 1997, aged 18, Gareth left home to begin a PhD at Manchester University, acquiring an advanced level of learning that sat uneasily with his elementary life skills, although his academic mentor, professor Terry Hewitt, believed he could take it: "He was very private and worked well on his own. He was certainly not a 'geek' even if he was reserved." His dissertation focused on computer games. They had always been a private passion, with him achieving an online reputation as unbeatable. Three years on, aged 21, Gareth was approached by the British security services, who the university believes had spotted his precocious online gaming abilities. Hewitt recalled: "In 2000, Gareth came into my office and said, 'I'm going to GCHQ'. He had to tell me, as I was his referee for his vetting." Gareth left Manchester and Anglesey behind to sign the Official Secrets Act.

Williams began an advanced maths course at St Catharine's College, Cambridge, but withdrew the following year, torn between academia and the demands of his new employer. At GCHQ, he was seen as stable, reliable and potentially brilliant, according to a contractor who worked there at the time. Speaking on condition of anonymity, he recalled that GCHQ was a world in which Gareth fitted, working alone, or in small teams, waist-deep in mathematical problems, challenged by cutting-edge technology and real-time problems. Possibly for the first time, he was part of a community of similarly focused individuals who were taken on for their discretion as well as for their brains, intercepting, monitoring and analysing email and telephone calls from around the world. He did feel the pressure, constantly, according to the contractor, who noted that at GCHQ many technicians were as clever or cleverer than Williams.

Work became his life. There was little time for much outside it, something evidenced by how he chose to live, renting a modest room in the Prestbury area of the town. Inside was a single, child-sized bed, a small chest of drawers, three upright wooden chairs and a gas heater – all furnished by Jenny Elliot. Over the decade he spent there, Williams introduced no luxuries, accumulated no clutter. He brought no one back. He rarely even went out after work, excepting an occasional "drink with colleagues". He worked late into the night and rose early. There was something oppressively austere about him, those colleagues thought, something they used to call his "GDR sensibility". So serious and quiet was he that when he did become animated, he "alarmed people with his unusual laugh".

Due to the nature of his work, it is hard for those who met Gareth to describe what he did, and those who talked to us did so having already left GCHQ. But all agreed that, although very capable, Williams was "a middle-ranking" technician "asserting himself in a number of sensitive areas, any one of which could theoretically have brought him into contact with enemies of our state". While GCHQ gathers intelligence from Europe, Africa and Russia, its partners in the US National Security Agency (NSA) also had access to British and European signals traffic at Menwith Hill, a top-secret RAF base near Harrogate, in Yorkshire.

When Williams arrived at Menwith, shortly after the invasion of Iraq in March 2003, GCHQ and the NSA were increasingly focused on the threat of Islamist violence at home and abroad. "Between the UK and the US, we were deluged by the chatter from radicals here and over the Atlantic as they talked on landlines, satellite or mobile phones – with contacts in hot countries like Pakistan and Afghanistan, as well as with Saudi, and hubs like Dubai," one former GCHQ security consultant said. "A group around Williams worked on software to assemble, search and analyse the data, drawing out patterns and meanings."

Williams's elevation within GCHQ came with the "Liquid Bomb" plot, where a group of British radicals of Asian origin were found to be planning to detonate home-made explosives on board seven flights to major north American cities. Intercepting emails and phone calls between these plotters and their contacts abroad, Williams flew between the UK and the US, working at Fort Meade (the NSA HQ, in Baltimore).

Williams was brought into close proximity to US intelligence, Islamic radicals and Middle Eastern agents. He would rub shoulders with the Russians, too, according to a foreign intelligence analyst based in the UK, who described how technology and software honed by GCHQ was deployed in tracking a Moscow-backed sleeper cell to which Britain had been alerted as early as 2003. The case blew up in June 2010 when 10 people were arrested in the US and accused of being part of an espionage ring. One of them, a glamorous 28-year-old called Anna Chapman, had lived for five years in the UK. According to the former GCHQ contractor, "Williams had been obsessed by the case, its methodology and characters."

He was now on secondment to MI6, with its focus on foreign targets and intelligence-gathering, becoming more of a spy than a desk-bound technician, although Williams would shortly be called back to GCHQ to work on a new and ambitious project.

He had been liaising again with counterparts at the NSA, who were part of an effort to create an American cyber defence policy, to prevent the siphoning off of secret or commercially sensitive data or a military style assault on defence or civil systems. Its importance was alluded to last month in a speech by home secretary Theresa May, in which she identified cyber crime as among the pre-eminent future threats facing the UK. A cyber crimes specialist who knew Williams revealed that at the time of his death he was researching British vulnerability to Russian, Turkish and Chinese gangs: "He was already on top of it."

Rogue individuals and nation states, Islamist terror groups and radical loners, extortionists and organised criminals – these were just some of those Williams had observed, investigated, disrupted and provoked. Any one of them was capable of reciprocating, lethally.

Back at the Yard, Operation Finlayson detectives were looking at a vast range of potential threats, but still had few clues, no accomplices or even a plausible denouement for a man whose personal and professional inclinations were to camouflage everything he did.

By September, an exhaustive trawl had come up with some CCTV footage. In the frames released to the public, Gareth Williams saunters by Holland Park tube station in west London at 3pm on 14 August, three days after he had last spoken to his sister.

Staff at the nearby Valerie's Patisserie said that in the week before, a man fitting Gareth's description had spent several hours in the cafe on consecutive days, sitting at the rear with a laptop, receiving occasional visitors. One security source said that he believed Williams was "working" in west London – monitoring one of the many foreign embassies in the area – just as he had been "working" while on his recent "holiday" to the US.

Police released footage of Williams on 15 August, too. At 2pm, he was filmed at a cash machine, visiting Harrods, walking at 2.30pm via Hans Crescent in Knightsbridge towards Sloane Street. Soon after he vanished.

By the end of September, after a second barrage of toxicology tests proved inconclusive, Gareth Williams's body was released and laid to rest in North Wales. On 26 September, the extended Williams clan gathered outside the Bethel chapel in Holyhead to bid farewell. Given the florid rumours that had circulated around his death, they were surprised by the appearance there of Sir John Sawers, head of MI6. Williams had been with his service for less than a year, yet Sawyers told reporters, "I wanted to be here today as the only public face of the Secret Intelligence Service. My deepest sympathies go to the family. Gareth was a hugely talented person, and he was very modest and generous as well. He did really valuable work with us in the cause of national security."

What a difference 30 days make. After an investigation that had failed to turn up the smallest vice, Williams was no longer lampooned but celebrated as a very British spy: modest, capable and known well by no one.

As they buried Gareth, Dr Chris Subbe, speaking for his wife Ceri and the family, recalled how they had laughed and chatted with him at the Ritz, celebrating their marriage and his brilliant career. "The world was ours for the taking," he said. "Yet here I am three months later to the day, trying to describe your rich life with my poor words."

• This article was amended on Wednesday 24 November 2010. We incorrectly referred to Dr Chris Subbe as a senior registrar at Wrexham Maelor Hospital. Dr Subbe is a consultant at Wrexham Maelor Hospital. This has been corrected.