This is the story of a piece of paper no bigger than a credit card, thrown away in a dustbin on the Heathrow Express to Paddington station. It was nestling among chewing gum wrappers and baggage tags, cast off by some weary traveller, when I first laid eyes on it just over a month ago.

The traveller's name was Mark Broer. I know this because the paper - actually a flimsy piece of card - was a discarded British Airways boarding-pass stub, the small section of the pass displaying your name and seat number. The stub you probably throw away as soon as you leave your flight.

It said Broer had flown from Brussels to London on March 15 at 7.10am on BA flight 389 in seat 03C. It also told me he was a "Gold" standard passenger and gave me his frequent-flyer number. I picked up the stub, mindful of a conversation I had had with a computer security expert two months earlier, and put it in my pocket.

If the expert was right, this stub would enable me to access Broer's personal information, including his passport number, date of birth and nationality. It would provide the building blocks for stealing his identity, ruining his future travel plans - and even allow me to fake his passport.

It would also serve as the perfect tool for demonstrating the chaotic collection, storage and security of personal information gathered as a result of America's near-fanatical desire to collect data on travellers flying to the US - and raise serious questions about the sort of problems we can expect when ID cards are introduced in 2008.

To understand why the piece of paper I found on the Heathrow Express is important, it is necessary to go back not, as you might expect, to 9/11, but to 1996 and the crash of TWA Flight 800 over Long Island Sound, 12 minutes out of New York, with the loss of 230 lives. Initially, crash investigators suspected a terrorist bomb might have brought down the aircraft. This was later ruled out, but already the Clinton administration had decided it was time to devise a security system that would weed out potential terrorists before they boarded a flight. This was called Capps, the Computer Assisted Passenger Pre-screening System.

It was a prosaic, relatively unambitious idea at first. For example, in highly simplistic terms, if someone bought a one-way ticket, paid in cash and checked in no baggage, they would be flagged up as an individual who had no intention of arriving or of going home. A bomber, perhaps.

After 9/11, the ambitions for such screening grew exponentially and the newly founded Department of Homeland Security began inviting computer companies to develop intelligent systems that could "mine" data on individuals, whizzing round state, private and public databases to establish what kind of person was buying the ticket.

In 2003, one of the pioneers of the system, speaking anonymously, told me that the project, by now called Capps II, was being designed to designate travellers as green, amber or red risks. Green would be an individual with no criminal record - a US citizen, perhaps, who had a steady job and a settled home, was a frequent flyer and so on. Amber would be someone who had not provided enough information to confirm all of this and who might be stopped at US Immigration and asked to provide clearer proof of ID. Red would be someone who might be linked to an ever-growing list of suspected terrorists - or someone whose name matched such a suspect.

"If you are an American who has volunteered lots of details proving that you are who you say you are, that you have a stable home, live in a community, aren't a criminal, [Capps II] will flag you up as green and you will be automatically allowed on to your flight," the pioneer told me. "The problem is that if the system doesn't have a lot of information on you, or you have ordered a halal meal, or have a name similar to a known terrorist, or even if you are a foreigner, you'll most likely be flagged amber and held back to be asked for further details. If you are European and the US government is short of information on you - or, as is likely, has incorrect information on you - you can reckon on delay after delay unless you agree to let them delve into your private details.

"That is inconvenient enough but, as we tested the system, it became clear that information was going to be used to build a complete picture of you from lots of private databases - your credit record, your travel history, your criminal record, whether you had the remotest dubious links with anyone at your college who became a terrorist. I began to feel more and more uncomfortable about it."

Eventually, he quit the programme.

All of this was on my mind as I sat down with my computer expert, Adam Laurie, one of the founders of a company called the Bunker Secure Hosting, to examine Broer's boarding-pass stub. Laurie is known in cyber-circles as something of a white knight, a computer wizard who not only advises companies on how to make their systems secure, but also cares about civil rights and privacy. He and his brother Ben are renowned among web designers as the men who developed Apache SSL - the software that makes most of the world's web pages secure - and then gave it away for free.

We logged on to the BA website, bought a ticket in Broer's name and then, using the frequent flyer number on his boarding pass stub, without typing in a password, were given full access to all his personal details - including his passport number, the date it expired, his nationality (he is Dutch, living in the UK) and his date of birth. The system even allowed us to change the information.

Using this information and surfing publicly available databases, we were able - within 15 minutes - to find out where Broer lived, who lived there with him, where he worked, which universities he had attended and even how much his house was worth when he bought it two years ago. (This was particularly easy given his unusual name, but it would have been possible even if his name had been John Smith. We now had his date of birth and passport number, so we would have known exactly which John Smith.)

Laurie was anything but smug.

"This is terrible," he said. "It just shows what happens when governments begin demanding more and more of our personal information and then entrust it to companies simply not geared up for collecting or securing it as it gets shared around more and more people. It doesn't enhance our security; it undermines it."

Just over $100m had been spent on Capps II before it was scrapped in July 2004. Campaigners in the US had objected to it on grounds of privacy, and airlines such as JetBlue and American faced boycotts when it emerged that they were involved in trials - handing over passenger information - with the Department of Homeland Security's Transportation Security Administration. Even worse, JetBlue admitted it had given the private records of 5 million passengers to a commercial company for analysis - and some of this was posted on the internet.

But the problems did not end with the demise of Capps II. Earlier that month, after 18 months of acrimonious negotiation, the EU caved in to American demands that European airlines, too, should hand over passenger information to the United States Bureau of Customs and Border Protection, BCBP, before their aircraft would be allowed to land on US soil. The BCBP wanted up to 60 pieces of information routinely gathered by booking agencies and stored as a Passenger Name Record, PNR. This included not only your flight details, name, address and so on, but also your travel itinerary, where you were staying, with whom you travelled, whether you booked a hire car in the US, whether you booked a smoking room in your hotel, even if you ordered a halal or kosher meal. And the US authorities wanted to keep it all for 50 years.

At first, the European Commission argued that surrendering such information would be in breach of European data protection law. Eventually, however, in the face of huge fines for airlines and cancelled landing slots, it agreed that 34 items from PNRs could be handed over and kept by the US for three and a half years.

Capps II was superseded by a new system called Secure Flight in August 2004. Later, in October last year, the BCBP demanded that airlines travelling to, or through, the US should forward "advance passenger information", including passport number and date of birth, before passengers would be allowed to travel. It called this the advance passenger information system, or APIS. This is the information that Laurie and I had accessed through the BA website.

"The problem here is that a commercial organisation is being given the task of collecting data on behalf of a foreign government, for which it gets no financial reward, and which offers no business benefit in return," says Laurie. "Naturally, in such a case, they will seek to minimise their costs, which they do by handing the problem off to the passengers themselves. This has the neat side-effect of also handing off liability for data errors.

"You can imagine the case where a businessman's trip gets delayed because his passport details were incorrectly entered and he was mistaken for a terrorist. Since BA didn't enter the data - frequent flyers are asked to do it themselves - they can't be held responsible and can't be sued for his lost business."

By the time I found the ticket stub and went to Laurie, he had already reported his suspicions about a potential security lapse to BA (on January 20) by email. He received no response, so followed up with a telephone call asking for the airline's security officer. He was told there wasn't one, so he explained the lapse to an employee. Nothing was done and he still has not been contacted.

Three months ago, after further objections in the US, but before our investigation, Secure Flight was suspended after costing the US taxpayer $144m. At the time, Kip Hawley, transportation security administrator, said: "While the Secure Flight regulation is being developed, this is the time to ensure that the Secure Flight security, operational and privacy foundation is solid."

The TSA said it would continue its passenger pre-screening programme in yet another guise after it had been audited and added that it had plans to introduce more security, privacy and redress for errors - confirming critics' suspicions that no such systems were yet in place. To the consternation of privacy activists in Europe, the TSA also spelled out plans for its desire for various US government departments to share information, including yours and mine.

Dr Gus Hosein, a visiting fellow specialising in privacy and terrorism at the London School of Economics, is concerned about where the whole project will go next.

"They want to extend the advance passenger information system [APIS] to include data on where passengers are going and where they are staying because of concerns over plagues," he says. "For example, if bird flu breaks out, they want to know where all the foreign travellers are. The airlines hate this. It is a security nightmare. Soon the US will demand biometric information [fingerprints, retina scans etc] and they will share that around.

"But what the BA lapse shows is that companies cannot be trusted to gather this information without it getting out to criminals who would abuse it. The potential for identity theft is huge, but the number of agencies among which it will be shared is just growing and growing."

And that is where concern comes in over the UK's proposed ID cards, which may one day be needed to travel to the US. According to the Home Office, the identity cards bill currently going through Parliament allows for up to 40 pieces of personal information to be held on the proposed ID card, with digital biometric details of all of your fingerprints, both your irises and your face, all of which can be transmitted to electronic readers. The cards will contain a microchip the size of a grain of sand linked to a tiny embedded antenna that transmits all the information when contacted by an electronic reader.

This readable system, known as Radio Frequency Identification, or RFID, has recently been installed in new British passports. The Home Office says the information can be transmitted across a distance of only a couple of centimetres because the chips have no power of their own - they simply bounce back a response to a weak signal sent from passport readers at immigration points.

However, the suspicion is that the distance over which the signal can be read relates only to the weakness of the signal sent out by the readers. What if the readers sent out much stronger signals? Potentially, then, criminals with powerful readers could suck out your information as you passed by. The Government denies that this scenario is viable, but, in January, Dutch security specialists Riscure successfully read and de-encrypted information from its country's new biometric passports from a distance of about 30ft in just two hours.

"The Home Office says British passport information is encrypted, but it's a pretty basic form of encryption," says Hosein. "Everyone expects the ID cards to be equally insecure. If the government insists they won't be cracked, read or copied, they're kidding themselves and us."

BA has now closed its security loophole after being contacted by the Guardian in March, but that particular lapse is beside the point. Because of the pressure being applied to airlines by the US, breaches will happen again elsewhere as our personal data whizzes around the globe, often without our knowledge or consent.

Meanwhile, accountability remains lamentable. Several calls to the US Transportation Security Administration were not returned.

Perhaps the last word should go to Mark Broer, the man whose boarding pass stub started off this virtual paper chase. He is aged 41 and is a successful executive with a pharmaceutical recruitment company. When I told him what we had done with his boarding pass stub, he was appalled.

"I travel regularly and, because I go to the US, I submitted my personal information and passport number - it is required if you are a frequent flyer and want to check yourself in," he says. "Experienced travellers today know that they have to give up information for ease of travel and to fight terrorism. It is an exchange of information in return for convenience. But as far as I'm concerned, having that information leaked out to people who could steal my identity wasn't part of the deal."