It was 8.30 on a gloomy winter morning – Thursday 11 December – when the FBI's Agent Theodore Cacioppi made a house call. A penthouse call, to be exact. Accompanied by a sidekick from the bureau, Cacioppi strode into the lobby of the graceful pre-war building on Manhattan's monied Upper East Side, and announced himself to the doorman who guards access to the elite residents inside. The relatively modest lobby, with leather chairs and a single orchid for decoration, is at the opposite end of the scale to the dripping opulence of a Donald Trump creation. The giant apartments at 133 East 64th Street – just two per floor – start at $5m, and duplex penthouses at $7m. It's an address for people who want to enjoy the luxuries great wealth affords without shouting about it.

As Cacioppi was ushered to the elevator he cannot have realised that he was about to unmask the greatest fraudster the world has ever seen. He was there not because of years of dogged detective work, but to follow up an emotional confession made the previous evening to the suspect's sons, who had turned their father in. At 8.30am, Bernard Lawrence Madoff let the officers in.

One of the most outwardly respectable gentlemen in New York finance for decades, Madoff confessed to conning his family, friends, colleagues, clients and regulators to the tune of $50bn. With his confession to Cacioppi, Madoff drove a final nail into the coffin of the dazzling, wild Wall Street of the past quarter of a century.

Download the new Independent Premium app Sharing the full story, not just the headlines

"We are here to find out if there is an innocent explanation," Cacioppi said quietly. The 70-year-old financier – his grey hair receding but thick and kept long, swept back; his skin tanned, making him look younger – paused, then replied. "There is no innocent explanation."

The words unleashed a firestorm. As piece by piece the breathtaking scale of this fraud came to light, revealing a vast network of rich and famous victims across the world and banks brought to the brink of collapse, it has become increasingly mind-boggling to trace this maelstrom back to Bernard Madoff. How could one man alone evaporate so much wealth? What was he thinking? It is hard to imagine that we will ever have more than a partial answer.

Enron was at least a corporation, where a cadre of executives spun a deceit whose spectacular end came to symbolise the excesses of the dot.com era. The Madoff affair (it is indeed pronounced "made off") is a more baffling concept.

The bare bones are easily understood. He was lying when he said he was investing his clients' money and generating enviable, steady returns. There were no profits; instead, their money was going to pay other clients who wanted to cash out. It is an old fraudster trick, a pyramid scheme, or "Ponzi scheme", to use the American term, after the immigrant who conned New York's Italian community around 1920. Charles Ponzi discovered that you need ever increasing amounts of new cash to keep the game going; what is so astonishing is how, over so many years, Madoff did manage to harvest those ever-increasing sums. Every month, he sent out statements telling clients how their nest eggs were stacking up, statements whose fictitious balances totalled about $50bn, as he told Agent Cacioppi that morning. In fact, there was less than $300m in the pot.

It was just after the end of the trading day on Wall Street when the details of that morning's arrest popped up on the newswires, and CNBC, the business channel that blares out from the corner of most offices made its breathless announcement: "If you work on a trading desk, stop what you're doing for one second before you walk out the door and clean your desk out for the day," said presenter Michelle Caruso-Cabrera. "Bernie Madoff has been arrested...."

Madoff may not have been a household name, but for almost 50 years, Madoff Investment Securities has been one of the industry's most prominent trading firms and Madoff one of its more famous faces. His is just the sort of hardscrabble tale that Wall Street loves to mythologise – the boy from Queens, New York, who saved $5,000 from selling sprinkler systems and then working as a lifeguard on Long Island, and used the money to set himself up as a trader of penny stocks. Madoff himself revelled in his own myth. A statue in the grounds of his Palm Beach mansion is of two lifeguards staring out to sea. (We learnt this because last month, some scurrilous thief stole it, only to return it days later with a note attached: "Bernie the Swindler, Lesson: Return stolen property to rightful owners. Signed by – The Educators.")

Madoff's mother also appears to have been a stockbroker, albeit one who fell foul of regulators, who accused her of failing to comply with financial reporting rules in the 1960s. Bernard was born to Sylvia and Ralph Madoff in the tight-knit, mainly Jewish community of Laurelton, Queens on 29 April 1938, and appeared to make little academic impression at Far Rockaway High School or later at the University of Alabama, or Hofstra University, back in Long Island, where he got a degree in political science.

Daily coronavirus briefing No hype, just the advice and analysis you need Enter your email address Continue Continue Please enter an email address Email address is invalid Fill out this field Email address is invalid Email already exists. Log in to update your newsletter preferences Register with your social account or click here to log in I would like to receive morning headlines Monday - Friday plus breaking news alerts by email Update newsletter preferences

But Madoff Investment Securities, which he founded in 1960, was one of the iconoclastic firms who challenged the New York Stock Exchange's old-school brokers, using aggressive marketing techniques to win clients, and promoting a new era of electronic trading. Madoff became something of a spokesman for these outsiders, and helped create the rival Nasdaq electronic stock exchange, where he later served as chairman. Although many people who came into contact with him on the golf courses of Florida, have described him as aloof, he was clubbable with the right people, and chummy with the finance industry's regulators at the Securities and Exchange Commission, supposedly consumers' sharp-toothed watchdog on Wall Street. They would rib each other – Madoff laughingly called enforcers at the SEC "the enemy" when his niece, the glamorous Shana Madoff, married one of them – but the SEC relied on him and other members of his family to advise them on ways to modernise the markets.

Madoff's wife, Ruth, meanwhile, was evolving into the New York society wife. Two years his junior at Far Rockaway, Ruth Alpern had been his childhood sweetheart, an outgoing blonde who was always one of the most popular in her class. At times, she kept an office at the firm, and investigators believe her name is on some transactions and that she may have helped to reconcile the business's bank accounts, though there is no evidence she knew of her husband's fraud. Indeed, the topic of whether she could have been at her husband's side for 50 years without believing something was amiss is now a staple of daytime talk shows. She has not been seen in public since her husband's arrest.

Ruth Madoff mainly busied herself with philanthropic ventures, particularly the couple's charitable foundation, set up in 1998 and a big giver to cancer causes and the arts. In 1996, she executive-edited a cook book, Great Chefs of America Cook Kosher, a vanity project. She would spend time decorating the couple's numerous properties, such as the apartment in Cap d'Antibes on the French Riviera, and in Florida, where their $9.4m winter mansion has five bedrooms and a boat dock for the yacht, The Bull. They joined all the right clubs, moved in all the right circles.

So it is many years since Madoff could have been described as a Wall Street outsider; by the time of his confession, he appeared the grandee. Established, trustworthy. Maybe it was keeping up those appearances that led him into his desperate deceptions, perhaps after an unexpected loss that he hoped to recoup later. Or maybe the deceits began long before that, rooted in those raw early days of trading and the desire to build the business that his mother could not. Investigators are still only at the start of their work.

Appearances are important to Madoff, and employees have numerous tales that suggest an obsessive compulsive disorder. A former secretary has described how he remodelled offices to eliminate semi-circular shapes, detailing how he squared off the interior of his New York office which had moved into the desirable, elliptical skyscraper nicknamed the Lipstick Building. Other stories tell of how he installed CCTV to monitor employees in the London branch (which purportedly managed only a small chunk of Madoff's personal wealth but is at the centre of an inquiry into whether he pushed money offshore). Any chipped paint would have to be filled in with marker pen. He once ripped out a carpet tile that had a pear juice stain on it, and rushed to replace it with a clean spare. The Madoff Investment Securities colour scheme of black and grey was taken to such an extreme that it took on a Soviet feel. There was a strict ban on using blue pens. Black ink only.

Nowhere was Madoff's bizarre behaviour on display more clearly than on the day, a week after his arrest, when the public got the first glimpse of him returning to the Upper East Side after a courthouse tussle over his bail. Of all the things that he was wearing as he marched along the side side of his building, pushing back a swarm of photographers, it wasn't the black diamond-quilted jacket and the baseball cap that caused comment – it was the fixed grin. And if there is anything calculated to cause public outrage, it is the sight of a self-confessed fraudster grinning. Suddenly, Bernard Madoff was public enemy No 1.

At future hearings, where his bail terms have been tightened, he would be escorted to and from the courthouse by US marshals, as protesters circled. One man, camped outside the building where Madoff has been under house arrest for more than a month, angled a placard so it could be seen from the penthouse window: "Bernie, it's not too late to do the right thing. JUMP!" Amid persistent rumours that Russian mob money found its way into the Madoff Ponzi scheme, a panic button has been installed in the apartment, along with 24-hour surveillance, as much to protect him as to check he is not planning to flee. He wears a bullet-proof vest for trips to court.

The fraudster's notoriety is such that trinkets such as pens and rugs and other giveaways emblazoned with the Madoff Investment Securities logo are selling for hundreds of dollars on eBay. A hot sauce called "Bernie in Hell" has been launched by a New York artist. The New York Times, at the weekend, compared him to a psychopath and the Seventies serial killer Ted Bundy.

Madoff never murdered anyone, but the distress he leaves in his wake is huge. Numerous celebrities have given his crime a photogenic face, among them Kevin Bacon, Steven Spielberg and his business partner Jeffrey Katzenberg, and Zsa Zsa Gabor, down a reported $7m. But it goes much, much further. The children of the Long Island Jewish set who relied on Madoff money for their trust funds will probably survive. But baby-boomers who had budgeted on those fictitious monthly accounts to pay their nursing care and medical bills through retirement face a suddenly bleak old age. Ordinary pension funds were among those whose billions were funnelled through intermediaries into Madoff, leaving Connecticut's firefighters among those hit. And the fraud has had a devastating effect on philanthropic enterprises – from human rights organisations rehabilitating former prisoners, to bone marrow donor networks – because many Jewish charities trusted money to Madoff directly or to the unwitting accomplices who handed over their own hedge funds for him to run.

One woman, Sondra Wiener, 74, has put her $900,000 retirement home in Palm Beach up for sale, and her son, 50-year-old Charles, has done the same with the home he built in 1971 on Long Island. Wiener was close to tears when a New York Post reporter spoke to him outside the four-bedroom house. "It's a rotten thing, emotionally devastating to our entire family," he said.

It's not a good time to be selling your home, but that is not what makes this story so memorable. Sondra Weiner is Bernard Madoff's sister; Charles worked at the firm for 30 years. Madoff was scamming even those closest to him.

Investigators are turning their attentions now to Madoff's brother, Peter, seven years his junior, who – although he held no stake in the firm – was a deputy at Madoff Investment Securities, so indispensable that the pair often refused to travel together in case of an accident. Bespectacled and bow-tied, affable and erudite, Peter Madoff collects classic cars almost as enthusiastically as his brother collects watches. Like his brother, he split his time between Long Island and Palm Beach, where he was networker-in-chief. Peter was in charge of the historic stockbroking side of the company, which was always kept separate from the phantom investment business.

It was to Peter that Bernard's two sons notionally reported, as they rose through the ranks. Andrew Madoff, now 44, and brother Mark, 42, have worked with their father all their adult lives. Mark was more deeply enmeshed in designing flashy electronic trading systems. He sat on a string of company boards – although he had to slow down to battle lymphoma earlier this decade – and hobnobbed in the rarefied Core Club, behind an unmarked door a few blocks from the Lipstick Building. The $15,000-a-year club boasts Bill Clinton, private equity billionaire Stephen Schwarzman, Hollywood mogul Harvey Weinstein and people from the pinnacle of fashion and food on its membership list. Mark Madoff has just cancelled his.

Both he and Andrew were victims of their father, not his accomplices, their lawyer says. They had "no access to overall financial information about their father's firm". Mark took his money out of the investment arm to fund a divorce from his first wife in 2000 and both sons used outside investment firms to run their own private philanthropic foundations, shielding them from ruin, but Andrew had millions invested with his father.

There was no Hanukkah gathering for the Madoffs. Lawyers are standing in between them all. It was to his brother that Bernard Madoff first confessed his fraud, according to the tale told to Agent Cacioppi, and a day later to his sons, who turned him in. Now Andrew and Mark have been told not to talk to their father or their uncle while investigators pick through who knew what and when, and since all are at least witnesses in the case they must avoid pre-trial contact. There is no suggestion that the FBI has evidence family members knew anything until those confessions.

The FBI and SEC have camped out at Madoff's 17th-floor offices of that lipstick-shaped, salmon-coloured skyscraper. It was here, we know now, that Madoff kept numerous sets of books, and where the code to this extraordinary fraud may finally be deciphered. His sons told the FBI that his investment business was run there, away from the historic trading business carried out on the bustling floors above. He was "cryptic" about the operations on the 17th floor, they said, and there is still no clarity on when he began taking clients' money with the promise of investment.

He ran the investment arm side by side with a trusted lieutenant, Frank DiPascali, whose arrival at the office in jumpers and jeans belied a taut personality. He had been introduced to Madoff a year after graduating from a Catholic high school in Queens 33 years ago. He lives more modestly than his boss: only a five-bed, five-bath house with a pool and cabana on seven acres in the New Jersey suburbs. He's "a blue-collar guy, not a Wall Street master of the universe", says his lawyer, Mark Mukasey, "and he is devastated".

Notes of investigators' leaked interviews with DiPascali describe his answers as "evasive" and "incomprehensible", but he has not been accused of wrongdoing and his lawyer has declined comment on the extent of his knowledge. Investors were told that DiPascali was the man who carried out the complicated trading that was supposedly returning double-digit percentage profits, year in, year out.

Visitors to the 17th floor describe it in almost eerie terms, a place with banks of computers but hardly a trader in sight. Supposedly the hub of a multi-billion-dollar fund, there were barely 20 dedicated employees. The volumes of options trading that Madoff claimed to be doing never showed up on any public exchange or records from the stockbroking side of the company, and no other firm has said it traded with the Madoff's investment business. The FBI believes Madoff may never have properly invested any of the money entrusted to him.

Fifty-billion dollars: $50,000,000,000. No matter how you write it, it is a breathtaking number. Few Wall Street money managers achieve a business on that scale. Madoff did it while having to maintain a fiction of Truman Show scale. For many years, he hid from public view the fact that he was managing other people's money. Even after he registered as an investment adviser, at no point did anyone realise the scale of the funds that were pouring into the black hole on the 17th floor.

How could one man do it? The answer is he did not do it alone. A vast network of fundraisers was assembled among a financial aristocracy spanning three continents. Often secretly, they passed to him billions of dollars, in cash, from acquaintances. None of which is to say these people knew they were aiding and abetting a fraud. They profited handsomely from not asking. Madoff hadn't set himself up as a traditional hedge fund, and didn't charge standard outrageous fees of "two and twenty" – 2 per cent to manage the money and 20 per cent of profits. A string of Madoff's middlemen outsourced vast portions of their own business to Madoff and charged clients handsomely for profits he claimed to have earned them.

But their lives, too, changed at 8.30am on 11 December. One after another, these powerbrokers were forced to admit they had been duped. Now facing catcalls about their ignorance, and lawsuits alleging negligence, these are the blind accomplices who made Madoff.

Carl Shapiro met Madoff 48 years ago. Shapiro was a successful Boston businessman and investor – the city is studded with buildings named in his honour, as he fed some of the fortune from his clothing business into philanthropic gestures. "A friend asked me to meet him, maybe throw him a little business," Shapiro, now 95, told the local paper in Palm Beach the weekend after Madoff's arrest. He handed over a first cheque for $100,000, "and he did very well with it. That was the beginning."

The pair became close. Friends said Madoff was the son Shapiro never had. And in the same way that a mutual acquaintance had introduced the two, Shapiro quickly became a cheerleader for Madoff's investment talents. In Boston, and later in Palm Beach, he steered friends and acquaintances into Madoff Investment Securities, a miracle-grow fund that seemed to nudge a percentage point or two higher most months. Madoff never reported a down year. Even when the stock market was plunging, he reported solid gains. The strategy, Madoff would say vaguely, was to buy shares in blue-chip companies but take out derivatives to mitigate potential losses. Any further inquiry would be met with a pained look, as if someone had asked for the secret formula for Coca-Cola. Still, the returns tasted good, no one reported any problem cashing out, and Bernie had such respectable friends. There was no reason to be suspicious, and every reason to be fearful that Madoff might turn down your investment, leaving you with the harsh choice of a volatile stock market or mundane interest from the bank. In Palm Beach, second home to the East Coast's rich, being in is all. Being turned down was like social death. Madoff hardly needed to solicit investors; they clamoured introductions.

The Palm Beach Country Club was the centre of the financial earthquake. The grand, stucco-walled building sits, secluded behind trimmed hedges, on the Atlantic coast, and members can range over the challenging, perfectly manicured 18-hole golf course designed almost a century ago by Scotsman Donald Ross. Perhaps a third of the club's members had been ushered into the Ponzi scheme...Perhaps a third of its members had been ushered into the scheme via Shapiro and his son-in-law Robert Jaffe. Members' losses probably come close to the billion dollar mark. The Shapiros alone have lost $545m.

The club is the height of exclusivity. As well as the entry fee well into six figures, members are required to contribute to charity. Set up in 1959 by wealthy Jews who had been excluded by the Anglo-Saxon cliques of other clubs, it remains mainly Jewish. Madoff joined Shapiro with a membership in 1996, making local acquaintances there and at the nearby Boca Rio golf club.

It was Jaffe who rang his father-in-law on the day Madoff's arrest became news. "Turn on CNN," he said simply. The revelation was "a knife in the heart," Shapiro said.

Jaffe is a larger than life character. Driving about in his vintage MG, he flaunted the wealth Madoff was bringing him. "The clothing I wear is more cutting edge," he told the Palm Beach Daily News when it profiled stylish residents. "It's a few years ahead of the pack. Once you've had filet mignon, you don't want to go back."

Jaffe set up a company for funnelling investors' money to Madoff. Cohmad Securities – now under investigation by the Massachusetts attorney general, who has subpoenaed Jaffe – proved to be a model for numerous others.

Another of Madoff's blind accomplices was Ezra Merkin, one of the most prominent members of New York's Jewish community, who has spent a charmed life at the intersection of finance and philanthropy. His parents, Hermann and Ursula, had arrived in the US as refugees from Nazi Germany, made a fortune in shipping and on Wall Street, and bestowed millions on projects such as the Fifth Avenue Synagogue in Manhattan and Yeshiva University, where Merkin and Madoff would eventually serve as trustees. "It is not easy to stay on the sidelines while others are busy getting rich," Merkin once wrote – and he had no intention of doing so. Through his hedge fund, Ascot Partners, he raised several billions of dollars of client money which he funnelled towards the extraordinary Madoff, on whose profits he came to rely. Yeshiva University saw $112m wiped out, having let Merkin run its endowment fund. In all, $2.4bn of Merkin-related money has evaporated. The 54-year-old told his clients in a letter after Madoff's arrest that he had invested substantially all their money with the fraudster, that he was "shocked" and that he himself had "suffered major losses from this catastrophe". Their immediate response was one not of sympathy for a family fortune and a personal reputation ruined, but one of fury. What had he been doing to earn his multi-million dollar fees?

Merkin now sits amid a blizzard of lawsuits from investors who say they believe he was managing the money himself, most prominently from New York University, which lost its $3m investment in Ascot and which wants to stop him from liquidating what remains of the fund. In the early days of the New Year, he resigned as chairman of General Motors' car loans arm, GMAC.

The betrayal of Jewish charities and individuals by one of their own has brought a particular brand of condemnation. It has also caused a shudder through the community because of the flecks of anti-Semitism that can be found in some public responses to the scandal. You don't have to read too far into an internet discussion on the subject to see the stereotype of the unscrupulous Jewish financier raised anew.

Ponzi schemes are notoriously concentrated within so-called "affinity groups" – but from the Italian community scammed by Charles Ponzi onwards, they have been scattered across a rainbow of ethnic communities. And in any case, history's greatest Ponzi scheme has sowed its destruction far beyond the Jewish communities of New York and Palm Beach. The biggest single character in the cast list of blind accomplices is a native of Nashville, Tennessee. Walter Noel is patriarch to one of the most famous – and photogenic – families in the world of hedge funds, and the biggest fish in Madoff's poison pond. His share of responsibility for the $50bn in losses: $7.5bn. His annual fee on the money his Fairfield Greenwich investment business was outsourcing to Madoff: at least $250m a year.

The scandal has generated more column inches than the Noels ever managed before 11 December – and that was plenty. With five tall, slender, socialite daughters, all of whom have married well-bred men now working in the family firm, the Noels were gossip page favourites for years. "Glamazons", Vanity Fair magazine described them in a fawning 2002 portrait: "The Noel sisters of Greenwich, Connecticut, are turning tabloid-fodder sister acts (that is, Nicky and Paris Hilton) on their heads. In lieu of dancing on tables, the five Noel women have made a name for themselves by shoring up the virtues of a nearly extinct aristocracy. They're well educated and well-married, and they're raising a pack of well-behaved multilingual children while keeping their string-bikini figures intact." In the testosterone-fuelled world of hedge funds, getting the Noel family Christmas card was a coveted joy. At least until Christmas 2008.

Noel, now 78 but looking a dozen years younger, had been funnelling ever-increasing sums to Madoff for 20 years. The pair became richer and more successful as their businesses grew symbiotically. It was the Noel family enterprise, more than any other, that brought Madoff mayhem to South America, the Middle East and Asia. In Brazil, where Noel's wife, Monica, hails from a wealthy family with Swiss roots, Fairfield employed Bianca Haegler, a niece, as its representative. In Geneva, the private bank UBP marshalled its clients into Fairfield; UBP's London office is run by a family friend.

Meanwhile, one Noel son-in-law, Yanko Della Schiava, is based in Lugano, Switzerland, selling Fairfield Greenwich funds in southern Europe. Philip Toub, whose wedding to Alix Noel in Mustique in 1997 made the society pages, marketed the group's funds in Brazil and the Middle East.

A third, Andres Piedrahita, a Colombian-born banker, ran Fairfield's UK operation and was the most successful fundraiser. Since convincing his father to fund a trip around the world on a cruise ship to combine his university studies with making international contacts, he has lived in Palm Beach, New York, London and Madrid, and all the while keeping in touch with the rich of his native Colombia – where it seems some $200m of local wealth has evaporated with Madoff.

With point-men across the world, and news of Madoff's extraordinary investment performance spreading, wealthy individuals and supposedly sophisticated investment managers scrabbled for access. Austria is reeling from the nationalisation of one of its fastest-growing banks, Vienna-based Bank Medici. Its founder, a 60-year-old called Sonja Kohn – famous for the giant red wig she wore to adhere to her orthodox Jewish faith – denied reports that she had gone into hiding fearing Russian mobsters who may have been among those hit by her $3bn of losses. She said news of Madoff's arrest hit her like a "tsunami".

In the UK, Nicola Horlick, the fund manager dubbed Superwoman for her ability to juggle a career with raising children, saw shares in her latest fund venture plunge after it admitted putting almost 10 per cent of shareholders' assets – about £10m –with Madoff. Horlick knew who was responsible for her losses: US regulators. "I think now it is very difficult for people to invest in things that are meant to be regulated in America, because they have fallen down on the job," she told the BBC. It was a Frenchman who actually took responsibility squarely on his own shoulders.

On a dark Monday evening, 10 days after Madoff's arrest, René-Thierry Magon de la Villehuchet told the cleaners at his Manhattan office that he wanted them done by 7pm. After they left, the 65-year-old locked the door and returned to his desk. There he swallowed sleeping pills and placed a wastepaper basket beneath his desk to catch blood, before slashing his wrists with a box cutter. Staff found him the next morning, still sitting at his desk, the box cutter on the floor beside him. He had lost $1.4bn of his own and his clients' money, and the responsibility that he felt for their ruin eclipsed even the despair he felt over his own, according to friends and relatives.

One of the most prominent French financiers on Wall Street, a former head of Crédit Lyonnais Securities in the US, he was the scion of an aristocratic French family who would boast that 20 of his ancestors went to the guillotine. His hedge fund, Access International Advisors, enlisted intermediaries with links to the cream of Europe's high society and even royalty to garner clients, and those facing multi-million dollar losses included the continent's richest woman, Liliane Bettencourt, the L'Oreal heiress.

But not everyone trusted Madoff. Far from it. Harry Markopolos, a skinny, slighty nerdy accountant, was the resident maths genius in the offices of Rampart, a Boston investment firm that specialised in trading options. Rampart had heard of Madoff's enviable record back in 1999 and wanted a piece of the action, so Markopolos was handed the task of replicating the stock-and-options trading strategy that Madoff had vaguely but consistently outlined for his investors.

Except, even the maths genius couldn't do it; there is no way to guarantee positive returns every month irrespective of what the stock market is doing. Markopolos said there were only two conclusions. Perhaps Madoff was inflating returns by using an illegal trick called "front running", where the broker puts his own little bet on a share price first, knowing that his customers' order is about to push the price higher. More likely, as Markopolos wrote more than eight years before that confession last month, "Madoff Securities is the world's largest Ponzi scheme."

He passed his concerns via a contact at the local office of the SEC to the regulator's New York office. They said thanks, but nothing seemed to happen. Over the next few years, Markopolos would act as "an army of one", campaigning to unmask Madoff. Markopolos voraciously gathered facts, figures, rumour and innuendo and continued to deluge the SEC with tip-offs. In part, he hoped to win a reward (under US law, whistleblowers receive a proportion of any fines paid by misbehaving companies, and Markopolos has since turned himself into a freelance investigator encouraging and advising whistleblowers). Mainly, he just wanted to be proved right.

What Markopolos discovered during his campaign was that numerous market players were refusing to do business with Madoff because of the front-running rumours and other concerns. Many looked at the way Madoff's books were audited and shied away. It used a one-person audit firm, Friehling & Horowitz, based in a 13 foot by 18 foot office in a suburb miles outside New York. Saving pennies, Madoff would say, but the notion that this firm could audit a multi-billion dollar investment business struck many as ludicrous.

Several said that if Madoff was legit he would be running a formal hedge fund and pocketing the giant fees, rather than working only for trading commissions. Others had reached a similar conclusion to Markopolos, that the hedging system Madoff claimed to be using to safeguard himself from losses was implausible. It would require options trading on a scale that dwarfed anything going through the official exchanges.

A few were scared off by a 2001 article in the business magazine Barron's, the one time that the industry's scepticism was put into print, but Madoff was largely able to laugh it off. Markopolos was astonished to discover that sophisticated investors could tell him in one breath that they believed Madoff's returns were being faked, and then in the next that, hell, they were investing with him anyway.

Why did the SEC not put a stop to Madoff? Many believe the answer lies in the chumminess between Madoff and the regulators, whom Madoff himself advised. In one of his last acts as chairman of the SEC, Christopher Cox – the hapless former Congressman – ordered an investigation into all the organisation's dealings with Madoff, right down to the relationship between Madoff's niece, Shana, and Eric Swanson. Until 2006, Swanson was an SEC attorney in charge of overseeing Stock Exchange regulation of electronic trading and his team had carried out one of the investigations into Madoff Investment Securities. The pair started dating afterwards, his lawyers say.

Markopolos's allegations became more shrill until, in 2005, he got the full-on probe that he was seeking. His 19-page private submission to the SEC at that point is a masterclass in analysis, and it has been widely circulated on the internet since Madoff's confession. The title of his paper? "The world's largest hedge fund is a fraud."

Under this investigation, Madoff was finally forced to concede that his sideline in investment management had eclipsed the historic stockbroking business. He had been trying to shield the scale of his business from public view, keeping up the appearance that clients were entering an exclusive club, trying to keep below the radar. An angry SEC forced him to register formally as an adviser, and he declared client assets at that point of some $17bn. However, the regulator did not issue the public censure that could have alerted more investors of Madoff's shady behaviour, and never used the power of subpoena to recover the multiple sets of books kept under lock and key on the 17th floor. They should have paid closer attention to one of the passages in Markopolos's opus: "I've found that wherever there is one cockroach in plain sight, many more are lurking behind the corner out of plain view."

"If someone provides you with the wrong set of books, I don't know how you find the real books," Meaghan Cheung said, her eyes welling with tears when a reporter confronted her after the New Year. Markopolos has singled out Cheung, a branch head of the SEC's New York enforcement division at the time of the 2005 investigation, as the person responsible for the regulator's blunder. His comments sparked a media hunt for the 37-year-old mother of two, who quit the SEC last September. "Why are you taking a mid-level staff person and making me responsible for the failure of the American economy?" she said. "I worked very hard for 10 years to make a career, and a reputation, and that has been destroyed in a month."

By contrast, Markopolos's reputation has gone into the stratosphere. However, he is uncomfortable in the limelight, crying sick when Congress called him to testify at a hearing into Madoff. He has received offers from Hollywood, which wants to turn his crusade into a movie. So far, he has demurred. "They'll just add in sex and violence," he told The Boston Globe. "And why would people think I feel good about this?" Markopolos asked in a series of recent interviews. "People think I'm a hero, but I didn't stop him. He stopped himself."

The year 2008 will go down as one of the most brutal in investment history. Even good investments were in trouble, because investors had to pull profits to cover losses elsewhere. Madoff began to get more phone calls. Sorry, Bernie, but I have to take money out. The calls became a flood after the market carnage of September. The money that clients were demanding had long since been handed out to other investors on their way out, and the credit crisis meant new money was no longer flooding through the door. With a growing sense of panic, Madoff did what he had never before had to do: he began to solicit money.

Just before Thanksgiving, a pitch document was drawn up with details of a new fund, which he said he was raising from favoured clients, to take advantage of an oversold market. Ken Langone, a prominent New York investor, got the full charm. "You wouldn't think this guy had a trouble in the world," Langone said, but he demurred. He was turned off, he said later, by the "convoluted" monthly statements handed to investors and by the possibly illegal promise that investors in this new fund would be given preferential treatment over Madoff's other clients.

With redemption requests from clients now totalling $7bn, and barely a few hundred million in the pot, Madoff returned, desperately, to where it had all began 48 years earlier with a cheque for $100,000 – to Carl Shapiro. The men agreed an extraordinary final transaction, a payment to Madoff of $250m. It arrived around 1 December.

"Bernie seemed a little anxious this time," recalled Ruth Shapiro, 91, Carl's wife of 69 years. "He kept calling saying 'I didn't get it, it hasn't come yet, are you sure you sent it?'" The cheque would almost double the size of Shapiro's investment with his long-time friend. In all likelihood, he will never see any of it again.

The prosecutors will file formal charges. Until then, the best account of those final days comes to us via the FBI's Agent Cacioppi, as told to him by Andrew and Mark Madoff, who in turn say they are recounting what their father told them.

On Tuesday of that week, having decided the game was up, Madoff resolved to distribute what was left in the pot not to any of the clients demanding their money, but to employees in early bonuses. He penned $173m in cheques and placed them in a drawer. The next day, his sons, suspicious of their father's peculiar explanation – that the business had made a windfall on recent trading – demanded the truth. Madoff buckled. "I'm not sure I'll be able to hold it together," he told them. At the office, a few hours later, Madoff told them he had already confessed to his brother, Peter, that he was "finished", that he had "absolutely nothing", the business was "all just one big lie" and "basically a giant Ponzi scheme" which had been insolvent for years. He planned to surrender to the authorities the following week, but only after distributing the remainder of the funds to selected employees, family and friends.

Andrew and Mark listened – but did not agree. They called a friend for advice, Martin Flumenbaum, a lawyer, who called federal prosecutors and the SEC on their behalf.

Bernard and Ruth Madoff are still in the penthouse. He is under house arrest, on a $10m bail bond that his wife and brother struggled to afford. He has yet to utter a word in public.

A note he sent round to other residents at 133 East 64th Street reads: "Dear neighbors. Please accept my profound apologies for the terrible inconvenience that I have caused over the past weeks. Ruth and I appreciate the support we have received." So, an apology only for the media scrum, not for his alleged crimes. But, crucially for many victims, he has made no further apologies.

After the arrest, the Madoffs parcelled up Bernard's watches, cufflinks and jewellery and mailed it to relatives and close friends. Prosecutors became apoplectic; the victims' only hope is that Madoff's possessions and personal fortune will help them claw back a little of what they have lost. The trinkets have been returned, and must stay in the apartment under court diktat. But is it possible to see the attempted gifts as a kind of apology, for jobs lost, fortunes dented or evaporated, and trust shattered, at least to a small handful of people? Does Bernie Madoff feel guilt?

It could be years before the full story of Madoff's fraud can be told, but its epilogue is being written. He has become a potent symbol of a need to reform the way Wall Street behaves – and the way government watches Wall Street. Madoff is "a massive fraud that was made possible, in part, because the regulators ... dropped the ball," Barack Obama said last month. "And if the crisis has taught us anything, it's that this failure of oversight ... doesn't just harm individuals. It has the potential to devastate our entire economy."

Wall Street will no longer be allowed to govern itself. Even Alan Greenspan, high priest of laissez-faire, has said he was mistaken to believe investors could be trusted to rein in risky practices. Now we see it even more clearly with Madoff: everyone seemed to be trusting to everyone else to do the due diligence that should have been their own responsibility, even the wise men and the superwomen. A new era of harsher regulation is just around the corner. Wall Street, it seems, simply cannot be trusted.