On Christmas Day 1843, Joseph Gandy died in one of the windowless cells of a private asylum in Devon. He was the greatest perspective artist in the history of British architecture - dubbed "The English Piranesi" - and one of the great visionaries of the Romantic movement. No one knows where he is buried. Since the days of surrealism he has been a cult figure. His fantastical reconstructions of classical antiquity are collected by Ray Harryhausen, the brilliant designer of Jason and the Argonauts and Clash of the Titans.

The dungeon in which he died was a tragic echo of a dream he had as a young student of architecture. His account of that dream - which might have been written by Coleridge or Blake - is the key to understanding the tragedy of his life. In 1794 he was sailing to Italy, a young prodigy who had come top of the class at the Royal Academy. On board he lived the strict routine of the lonely traveller, dividing his day between study in his cabin and exercise on deck with a skipping rope. Two days after Land's End a storm blew up. Violently sick, he ate nothing but a raw egg and red wine. For supper, he had plum pudding and more wine. In bed he read Macbeth

In his dream he was standing on a staircase, watching a man sweet talk a girl at a window. Gandy was in love with the girl. Suddenly, a stranger took his arm. You must take her by force, he whispered; I am your friend, and will show you how. The man took him to palaces and dinners, and gave him presents. You are the Devil, Gandy exclaimed. He was seized, and woke in Hell. It was a city manufacturing luxuries in glass and gold, where "heavy cranes ... lift Foreign Goods from Ships". Hell was modern London.

Gandy escaped from "a low dungeon" by climbing a mile towards the light while the Devil drank with "lawyers, aldermen, &c". Outside was a mountain of empty wine bottles. Gandy made a sudden run up the slope, but the clatter of bottles alerted Satan. "I find you are like the rest of the world," he sneered: Ungrateful. But in a flicker of the eye, he relented and conjured up a circle of wine bottles to dance in the air around him: "Take a bottle of whatever you please and we will make it up again." Gandy took a bottle and hit the Devil. Pieces of glass stuck in his bald head. He hit him again, and escaped into the light.

What was the moral? Do not accept favours from the rich, he concluded. If you cannot repay them pound for pound, they will "play with the temper of a grateful mind ... debauch it, and make you their slave". Gandy's nightmare was about the artist selling his soul in return for patronage. He was poor, one of 12 children of a waiter at White's, the gentleman's club on St James's. In a stroke of luck, the club's architect saw his teenage sketches and took him as a pupil. The club's manager paid for his trip to Italy.

But he was also one of the first generation of artists who grew up to believe that self-expression should not be compromised by patrons or the public. In the 1770s and 1780s men such as Fuseli and Blake - the stars of Tate Britain's current Gothic Nightmares exhibition - transformed the image of the artistic genius. As the Tate exhibition shows, their hero was Prometheus, the Titan who in Greek legend was chained to the rock for stealing the gift of fire from the gods; fire was a metaphor for creativity, and the true artist was a rebel. But Gandy would discover that it was much harder for an architect to be a Romantic rebel than a poet or painter.

Three years in Rome deepened his absorption in the sublime of ruins, monuments and tombs. For seven hours he explored the Catacombs, scattering bran at his heels so that he could retrace his step through the labyrinth. He admired the mausolea of Roman heroes, which lined the Appian Way. Gandy returned to Britain penniless, but his talent was recognised by John Soane, the most original architect of the day. In January 1798 he joined Soane's office in Lincoln's Inn Fields, in the house that is now the museum. What Soane wanted was his ability to present an architectural design - built or unbuilt - in dramatic, luminous perspective. That year - and for 40 years afterwards - Gandy's paintings represented Soane's work at the Royal Academy exhibition. In an age before illustrated newspapers and photography, the annual exhibition of contemporary art and architecture - visited in six weeks by more than 50,000 of the richest people in Britain - was an architect's best opportunity to impress critics and potential clients. Gandy's watercolours made the other architects' drawings look like "dirty washings", wrote one newspaper.

The relationship between the two men was the most creative of its type in British architecture. Very quickly, Gandy understood Soane's unique style of decoration and his experiments in space. He shared his fascination with the use of top-lighting and coloured glass, the "lumière mystérieuse" - Soane's words - which cast a spell on the audience. After six months in the office Gandy painted a fantasy view of the Bank of England - Soane's proudest work - in ruins. The City of London is imagined as a swampy wilderness, as desolate as the Roman Forum in the dark ages. It is the earliest example in Europe of a drawing in which an architect imagines a structure he has built as a ruin. At one level, it is a meditation on the future of the British Empire. Babylon and Memphis, Carthage, Athens, and Rome ... why not London? But it also expressed Gandy's insight into Soane's mind.

To his clients Soane was the perfect professional, a punctilious, pragmatic workaholic. But in the privacy of his study he was insecure and melancholy, obsessed by the judgment of posterity. The painting was not shown to the public for decades; it is a highly personal meditation between master and disciple. "I respect you above myself," wrote Gandy. And, in turn, Soane paid Gandy to teach his son and heir to become an architect.

Gandy set up his own practice. Doric House in Bath and Storrs Hall on Lake Windermere show that he was as talented as any architect of his generation. But why did he build so little? One contemporary puzzled by his failure was John Constable. The Royal Academy received a letter in which Gandy described himself as coughing blood, and begged for support. Why, asked Constable, was a "genius" of architecture so neglected? He "was a bad-mannered man", his colleagues explained, "rude to any gentleman or nobleman who found fault with his designs ... he would not alter his drawings" to please a client.

Soane was able to live two lives. He could interrupt his poetic reveries when a client called; in the few steps between his private study and his office he could go from visions of ancient Athens to issues of plumbing, bricks, and percentages. He grew rich in a nation of shopkeepers, the favourite architect of the lawyers and aldermen of Gandy's nightmare.

Gandy, by contrast, refused to let clients visit the Soho lodging in which he lived with his wife and six children. When his solicitor called to explain his debts, Gandy wouldn't talk until he had finished his picture. He was thrown into the debtors' prison from where he wrote to Soane: "If Genius ... is placed in the lowest sink of vice and depravity, its sensibility is wounded to the utmost and with death it can only pass away." Soane paid Gandy's debts, and advised him to take a job in an office. If he became an architect's assistant, Gandy replied, "I should conceive myself a slave and debased below the character of man ... a non-entity."

What becomes evident, however, is that Gandy - in true Romantic fashion - saw his failure as the fault of society. Some critics agreed. When he exhibited his reconstruction of the ancient temple precinct of Jupiter, the Literary Gazette declared that Gandy was too good an architect for modern, commercial Britain. He should have been born in the age of Pericles, or Louis XIV.

It was the Guardian, no less, that most hurt Gandy. Reviewing his drawings of palaces at the 1821 exhibition, its critic said that anyone who proposed such "imaginative, impossible" project was not an architect but an illustrator. Gandy wrote five angry letters to the newspaper, the only time he put his ideas into words. Why should he not show the public the possibilities of architecture? Their imaginative horizons had been lowered by the drudgery of living in a city of commercial development. But London was the richest city in the world, the home of war heroes and great thinkers. Its architecture should reflect that.

In the 1820s he exhibited design after design of a royal palace to be built in Hyde Park at a cost of £3,000,000. It was not so much a residence for George IV as a temple for the British Empire. The drawings are dated "AD 2500": society - and Gandy - would be judged by posterity.

In 1838 - the year after Soane's death - Gandy exhibited at the RA for the last time. One picture was Architecture: Its Natural Model, a landscape in which an orang-utang builds a thatched hat. He had become obsessed by a project to illustrate the origins of architecture and to show that all the styles of the world were based on shared first principles. In its optimism and scope - and its interest in orangutangs - it is a project of the Enlightenment. But his second picture was a response to the dehumanised industrial age. A Design for a Cast-Iron Necropolis projected a future in which Londoners who died were inserted in metal cylinders, and slotted into cast-iron catacombs in the shape of giant pyramids.

The next year Gandy's family put him in the asylum in Plympton. It is heart-breaking to imagine the darkness of that windowless cell for a man so sensitive to light that he would stop to sketch a rainbow, or the blurry haze of sunrise on a frosty morning. His death was more anonymous than a metal cylinder. There is no record of his funeral, or his grave.

Brian Lukacher's book, Joseph Gandy, the result of 30 years' research, has inspired two exhibitions - in New York, and in London - which show that he was the greatest illustrator of architecture in British history. But it would be too easy to say that Gandy's struggle has been vindicated. The questions that his life raises about the relationship between the architect and society have yet to be answered, while the Gandy family upholds a tradition that his son Thomas started, forbidding his children ever to pick up a sketchbook. Posterity does not pay the bills. Gandy was the first true Romantic architect in Britain. And the last.

· Soane's Magician: The Tragic Genius of Joseph Michael Gandy is at the Sir John Soane's Museum, London WC2, until August 12. Tel: 020-7440 4246. The exhibition has been curated to coincide with the publication of Joseph Gandy: An Architectural Visionary in Georgian England by Brian Lukacher (Thames & Hudson).