It was the most famous house in Georgian England, but for some it was a sham and an architectural failure. Amanda Vickery considers its eccentric creator Horace Walpole

If you are an aficionado of architecture, you will know Horace Walpole as the creator of Strawberry Hill (1747-90), in Twickenham, west London, a flamboyant experiment in Gothic revival, forerunner of all those Victorian town halls, churches and stations which define our townscapes. You may also remember him as the author of The Castle of Otranto (1764), the first gothic novel, initiating a spooky literary genre still going strong.

Connoisseurs of Georgian culture recognise the voluble Walpole as a catty commentator on fashionable society. With 48 volumes of his correspondence in print, historians can rely on him for a gossipy opinion on most topics from adultery and chandeliers to wigs and Whigs. Hardly a party was thrown without Walpole on the sidelines taking sly notes for the amusement of posterity.

A new exhibition at the Victoria & Albert museum in London throws the spotlight on the peripheral observer and showcases the peculiarity of his taste. It restages Walpole's eclectic ­collection and evokes the dense ­interiors of his summer retreat, Strawberry Hill, as a curtain raiser for the reopening this autumn of the freshly restored house itself.

According to Michael Snodin, ­curator of the exhibition, Walpole "as a lively and incisive commentator shaped the way we see 18th-century politics and society. As the most ­important collector of his time he created a form of thematised historical display which prefigured modern ­museums. And Strawberry Hill was the most influential building of the early Gothic revival."

Walpole (1717-97) was the youngest son of Sir Robert Walpole, the first prime minister, and Catherine Shorter, daughter of a timber merchant. His parents were estranged even before his birth, the young Walpole remaining with his adored mother in the London house on Arlington Street in Piccadilly, avoiding Houghton – the Norfolk palace raised by his father as a monument to power.

After a conventional education (Eton and Cambridge topped off with the grand tour), Walpole became MP for Callington in 1743, where he never set foot. Effete and feeble, he bore little resemblance to his hearty father. Still he remained loyal to Whig politics and accepted sinecures worth £2,000 a year, bankrolling his "career" as a connoisseur and gentleman of leisure.

To us, Walpole appears decidedly peculiar – etiolated, fastidious and affected – and even in his own times he was considered singular. The writer Letitia Hawkins remembered a pallid aesthete tripping everywhere on his toes. "His figure was not merely tall, but more properly long and slender to excess: his complexion and particularly his hands of a most unhealthy paleness . . . he always entered a room in that style of affected delicacy, which fashion had then made almost natural . . . knees bent and feet on tip toe as if afraid of a wet floor."

Though Walpole had a penchant for the company of old ladies and un­marriageable or disgraced noblewomen, he evaded matrimony, remaining to his death aged 79 what used to be called a confirmed bachelor. Instead he drew about him a collection of highly cultured "dear friends"– men of sensitive taste but lesser background, who shared his obsessions. Walpole had an especially fraught and jealous relationship with Thomas Gray, of the famous "Elegy Written in a Country Churchyard", whom he met at Eton and took with him on his European tour.

Was Walpole gay? Is Strawberry Hill the manifestation of a gay aesthetic? The questions linger, even though searching for something akin to a ­modern homosexual identity is fruitless. Homosexual acts were criminal – sodomy was a capital offence – but virile men were known to take lovers of both sexes, while effeminate manners were seen as a Frenchified heterosexual weakness.

Walpole's biographers have often considered him effeminate and asexual, or at most passively homosexual. George Haggerty ponders the mystery again in the collection of essays that accompanies the exhibition. Walpole and his close male friends "did not identify themselves and were not identified by their contemporaries as sodomites, although several of them were known to feel desire for members of their own sex". Walpole's life-long correspondent, the Florentine expatriate Sir Horace Mann, was labelled a "finger-twirler" by the diarist and social commentator Hester Lynch Piozzi.

A romantic and erotic camaraderie is detectable among the aesthetes, archly expressed in interior decoration and antiquarianism. Anachronistically, but plausibly, Haggerty sees a camp sensibility at work. Strawberry Hill was to be the playground of affectation, a stage set on which Walpole performed his life, and an irresistible resort for his special friends.

Strawberry Hill was in fashionable Twickenham, a two-hour carriage drive from London, but enjoying some rays of royal glamour from nearby Richmond Palace and Hampton Court. The bosky Thameside bristled with the stately dowagers Walpole so admired, while the illustrious poet Alexander Pope had lived less than a mile away.

In 1747, Walpole leased a nondescript suburban house (built 1698) from Mrs Chevenix, a famous seller of trinkets. "It is a little plaything that I got out of Mrs Chevenix's shop and is the prettiest bauble you ever saw." In modern terms, the house is palatial, but by the standards of Georgian magnificence it was dinky.

"Lord God! Jesus! What a house!" cried Lady Townsend on an early visit. "It is just such a house as the parson's where the children lie at the end of the bed." As the second home of a fashionable gentleman who had a thoroughly classical headquarters in London, it was free from the rules governing the design of houses in town. Walpole set about Gothicising and extending, transforming the villa into "a gingerbread castle", "a Gothic mousetrap", a "paper house". He likened his adventure to that of Lord Burlington's pioneering of the neo-­Palladian in miniature at nearby Chiswick House: "As my castle is so diminutive, I give myself a Burlington air and say that as Chiswick is a model of Grecian ­architecture, Strawberry Hill is to be so of Gothic."

Walpole dated his interest in the Gothic from seeing King's College chapel as an undergraduate at Cambridge, constructed when "Art and Palladio had not reached the land nor methodised the Vandal builder's hand". He was hardly the first to pursue an antiquarian interest in British history or to admire the melancholy dignity of old cathedrals. The Gothic was seen as one decorative idiom among several, suited to informal rooms and garden structures. There was already a pseudo-Gothic summer house in Vauxhall pleasure gardens.

Plenty of nobles lived in crumbling houses, finding romance in heraldry and ancestry, old tapestry and stained glass. Gothic was a ready decorative choice for private chapels, especially for Catholics anxious to assert the continuity of the old religion. It also appealed to women with a strong sense of dynasty. The widow Lady Oxford began a fan-vaulted dining room at Welbeck Abbey in 1742, while Lady Pomfret built a castle-style house on Arlington Street in London (Walpole's own road) in the late 1750s.

Pretentious as he was, Walpole did not claim to have revived the medieval single-handed. He wrote to Mann in Italy for "any fragments of old painted glass, arms or anything", reassuring him of "the liberty of taste into which we are all struck". With papier-mâché friezes, Gothic-themed wallpaper, fireplaces copied from medieval tombs, a Holbein chamber evoking the court of Henry VIII, Dutch blue and white tiles on the floor, and modern oil paintings, china and carpets throughout, Strawberry Hill was hardly a faithful recreation of a medieval manor. Walpole wanted theatrical effect, atmosphere and "gloomth", not a time capsule. ­"Visions you know have always been my pasture . . . Old castles, old pictures, old histories and the babble of old ­people make one live back into centuries that cannot disappoint one."

The Gothic era he plundered seemed to encompass all the centuries before Inigo Jones (who transplanted the principles of Italian renaissance architecture under the patronage of Charles I). Any period from the dark ages to the Jacobean was ripe for plagiarism.

He made no doctrinaire claims. "I do not mean to defend by argument a small capricious house. It was built to please my own taste, and in some degree to please my own visions." Ever delicate, he admitted, "In Truth I do not mean to make my house so Gothic as to exclude convenience and modern refinements in luxury."

Teased by Mann as to whether the garden had to be medieval to match, he ruled "Gothic is merely architecture and as one has a satisfaction in imprinting the gloomth of abbeys and cathedrals on one's house, so one's garden on the contrary is to be­ ­nothing but riant [cheerful] and the gaiety of nature." But the garden still had to be picturesque. After a lecture on the ideal effect of the trees, the local ­nurseryman sighed: "Yes sir, I understand, you would have them hang down ­poetical."

Strawberry Hill was a confection, a mock-castle of a fake dynasty complete with a reproduction baronial hall, flourishing the arms and images of putative crusader ancestors on the ceiling. Through his mother, Walpole claimed descent from Cadwallader of Wales. The house became a museum to Walpole's expanding collection of art and relics, such as Queen Bertha's comb and the hair of Mary Tudor in a locket, though he was "outbid for ­Oliver Cromwell's night cap".

Snodin insists on Walpole's originality as a collector. He merged two pre-existing but distinct traditions – that of the virtuoso connoisseur seduced by art of all sorts, but also the antiquarian fascinated by historically significant objects (such as the spur King William drove into the flank of his horse Sorrel at the Battle of the Boyne). By 1797, Walpole had amassed at least 4,000 objects, not including scores of prints, drawings and books. The only things Walpole didn't collect were natural specimens and scientific instruments.

The diversity of Walpole's museum is recreated in the V&A exhibition, from 16th-century miniatures and sumptuous Reynolds portraits to Cardinal Wolsey's red hat. Which are Snodin's favourites? "For sheer glamour it has to be the gilded armour of Francis I, but one of the most curious objects is the black obsidian mirror used by the Elizabethan necromancer Dr John Dee to call up spirits, though Walpole didn't realise that it was originally used by the Aztecs in the human sacrificial rituals of their 'god of the smoking mirror'. We are still looking for some of Walpole's most famous objects, such as the jewelled dagger of Henry VIII."

Walpole wanted his objects to be ­admired. He gave personal tours to posh visitors, but left his housekeeper to herd the hoi polloi, for a guinea a tour. "'Tis the most amusing house I was ever in," remarked Lady Mary Coke, "so many pictures and things to help one to ideas when one wants a fresh collection; entertainment without company."

Walpole even produced a guidebook on his own printing press to initiate the cognoscenti, though inevitably he tired of traffic. "I keep an inn, the sign the Gothic castle," he moaned. "Never build yourself a house between London and Hampton Court. Everyone will live in it but you."

He introduced an advance booking system: "Every ticket will admit the Company between the hours of 12 and 3 before dinner. The house will never be shown after dinner nor at all but from the first of May to the first of ­October." And a final proviso: "They who have tickets are desired not to bring children."

Walpole was aggrieved to discover that visitors love to touch. "Two ­companies have been to see my house last week and one of the parties, as vulgar people always see with the ends of their fingers, had broken the end of my invaluable eagle's bill, and to ­conceal their mischief, had pocketed the piece."

At his death in 1797, the house passed to his cousin's unmarried daughter, Lady Anne Seymour Damer, a celebrated sculptor, and then to the Waldegraves, the family of his great niece. In 1842 the contents were sold off in the auction of the century, most never again seen together until now. (There was a small exhibition in 1980 with no international loans.)

Thanks to Walpole's publicity, Strawberry Hill was perhaps the most famous house in Georgian England, and inevitably fuelled voguish medievalism. But for Victorian purists such as Pugin, it was a sham. For modernists, it was an architectural failure of ghastly influence. BS Allen's Tides in Taste (1937) concluded that "reluctantly but inevitably one is reminded of the flocks of flimsy, starved houses that have sprung up since the war".

Wherever you stand on mock-Gothic, Strawberry Hill delivers un­rivalled access to both ideas and design. It is an exceptionally rich document – so rarely do original house, perspective views, objects, commentary and letters all survive.

For the architectural historian Charles Saumarez Smith, the house is important "not just as an oddity, much visited and admired, but because it was a presage of the way interiors would be used in the future, as a conscious instrument of personal expression, exploiting history to evoke a particular mood: Strawberry Hill was to become a private castle, an escape from time, a place of retreat."

Snodin agrees. "Walpole's cultural legacy was to pioneer a kind of imaginative self–expression in building, furnishing and collecting which still inspires us today. I suppose one of the take-home messages of the exhibition is: why not try it yourself?"

Horace Walpole's Strawberry Hill is at the Victoria & Albert Museum, London SW7 (020 7942 2000), from 6 March to 4 July. www.vam.ac.uk