One of the worst commercial decisions in history was made by Thomas Cadell, a London publisher, in November 1797. Cadell was new to the job. His father was a leading light in the publishing industry, a friend of Samuel Johnson and the publisher of Gibbon's Decline and Fall of the Roman Emperor; but the young Cadell was a tyro. So when he received a letter from one Rev George Austen, an unknown Hampshire clergyman, he wasn't impressed.

The Rev Austen told him that he had "in my possession a Manuscript Novel comprised in three Vols, about the length of Miss [Fanny] Burney's Evelina" and entitled First Impressions. What, he asked, would it cost him to get it published, and how much might Cadell be prepared to pay for the copyright? The reply came with frankly insulting speed. Austen's letter was sent back with the words 'Declined by Return of Post' scrawled across the top.

Like the Decca executive who turned down The Beatles in 1962 (saying "guitar groups are on the way out"), Cadell lived to regret his decision. For the Rev George had written on behalf of his daughter Jane, and the book First Impressions would eventually be published, 16 years later on 28 January 1813, with the title changed to Pride and Prejudice, one of the beacons of literary history. In the past 200 years, it has sold 20 million copies worldwide and never been out of print. Its first line is so well-known it's become a cliché. It is a truth universally acknowledged that Pride and Prejudice is the nation's favourite novel.

Download the new Independent Premium app Sharing the full story, not just the headlines

The story of the Bennet family's adventures in the marriage market of the early 19th century has been adapted for TV and the big screen umpteen times (Greer Garson played Lizzie in 1940 against a surly Laurence Olivier as Darcy; Keira Knightley and Matthew Macfadyen played them most recently in 2005), but has proved weirdly adaptable to different cultures and performance genres. There's been a Bollywood version (Bride & Prejudice), a Galilee-based Israeli TV mini-series, a Japanese comedy version (Hana Yori Dango), a Mormon-university update (Pride & Prejudice: a Latter-Day Comedy), a Broadway musical version of First Impressions and a concept album.

The book has been given dozens of metafictional rewrites and sequels (PD James wrote a murder mystery called Death Comes to Pemberley in 2011) including a notorious 'mash-up', Pride and Prejudice and Zombies, featuring cannibalism, ninjas and hordes of the walking undead. Lizzie Bennet has been given her own video blog (The Lizzie Bennet Diaries) and a TV mini-series, Lost in Austen, in which she swaps lives with a modern American woman, while Mr Darcy's side of the story has spawned a trilogy, by Pamela Aidan, called Fitzwilliam Darcy, Gentleman and an octet of spin-off books by Abigail Reynolds that include To Conquer Mr Darcy and What Would Mr Darcy Do?.

Jane Austen's publishing career began in 1810, when the publisher Thomas Egerton agreed to publish Sense and Sensibility – on commission, with Egerton holding the copyright. Jane's brother, Henry, and his wife (and cousin) Eliza jointly paid the printing costs. When the debut novel hit the streets on 31 October 1811, a newspaper advertisement coyly called it 'A New Novel by a Lady'. It was well reviewed, and much discussed in polite society, and netted its author £140. Egerton offered Jane £110 for the copyright to Pride and Prejudice.

It came out in January 1813, price 18 shillings, and was an instant hit. Richard Sheridan, the playwright, called it one of the cleverest things he'd ever read. George Henry Lewes, the critic and long-term partner of George Eliot, declared that he "would rather have written Pride and Prejudice, or Tom Jones, than any of the Waverley Novels," which was high praise at the time. A literary acquaintance of Jane's brother, Henry, assured him that it was "much too clever to be the work of a woman". Jane Austen herself stayed at home in Chawton, closeted with her mother, taking it in turns to read chapters of the book aloud to their poor spinster neighbour, Miss Benn. But her delight in what she called "my own darling Child" is evident in a letter she wrote to her sister Cassandra, which includes this tongue-in-cheek criticism:

"The work is rather too light & bright & sparkling; – it wants shade; – it wants to be stretched out here & there with a long Chapter – of sense if it could be had, if not of solemn specious nonsense – about something unconnected with the story; an essay on Writing, a critique on Walter Scott, or the history of Buonaparte – or anything that would form a contrast & bring the reader with increased delight to the playfulness and Epigrammatism of the general stile…"

What events, what concatenation of local circumstances, prompted her to write Pride and Prejudice – or First Impressions – in a stunning nine-month burst of activity between October 1796 and summer 1797? Many people attribute it to her meeting, and falling in love, with a young Irishman called Tom Lefroy. He and Jane were the same age – 20 – and met at a ball. He had completed a degree in Dublin, and was going to study for the Bar in London. At Christmas 1795, he had come to Hampshire to visit his aunt and uncle Lefroy at Ashe parsonage. And being handsome and charming as well as clever, he was invited everywhere. In the earliest of Jane's letters to survive, we read her breathless descriptions of a ball at Manydown House, the attentiveness of "this gentlemanlike, good-looking, pleasant young man" and how she gave "lessons" to the other dancers in showing a particular interest in a partner.

Displaying what Bridget Jones would later call "mention-itis", Jane brings up Lefroy's name again and again, mentions her visits to his uncle's parsonage, and rather scandalously invites Cassandra to imagine what they got up to, "everything most profligate and shocking in the way of dancing and sitting down together". She breaks off the letter to answer the door – and finds it is Mr Lefroy coming a-calling with a chaperone.

In all this activity – the ball, the dancing, the flirting, the (we assume) kissing, the writing, the visiting, the dreaming – Jane is as much the heroine as the spectator, says Claire Tomalin, Austen's most sympathetic biographer: "Jane is clearly writing as the heroine of her own youthful story, living for herself the short period of power, excitement and adventure that might come to a young woman when she was thinking of choosing a husband". But, of course, it was a Christmas romance that led nowhere. Their mutual attraction was evidently well-known locally; and Tom Lefroy was sent away before things became serious. There was an economic impulse behind his banishment: he was expected to become a barrister and the family breadwinner, and could not risk his life by marrying the daughter of a penniless clergyman. Jane never saw him after this Christmas visit.

But from then on, the feelings released inside her by this moment in love's spotlight also released her as a writer. "From now on," writes Tomalin, "she carried in her own flesh and blood, and not just gleaned from books and plays, the knowledge of sexual vulnerability: of what it is to be entranced by the dangerous stranger; to hope, and to feel the blood warm; to wince, to withdraw; to long for what you are not going to have and had better not mention. Her writing becomes informed by this knowledge, running like a dark undercurrent beneath the comedy."

Though Jane Austen was 37 when Pride and Prejudice was published, she was just 20 when she invented Lizzie Bennet – who is also, and forever, 20. It's easy to forget that the book concerns a sorority of teenagers and a succession of potential suitors – Bingley, Darcy, Wickham, Colonel Fitzwilliam – mostly in their twenties. Mrs Bennet has been portrayed in successive adaptations as a fusspot matron of mature years, but is barely into her forties, an immature, ditzy romantic who embarrasses her girls with her vulgarity. We can infer that Mr Bennet married her, without much thought to the future, when she was a sexy young minx and he was dazzled by her. Although a succession of adaptations has given the book a reputation for bonnets, minuets and genteel behaviour, the book is all about sex – and, of course, about the endless niceties of civilised discourse, the displays of accomplishments and minute calibrations of economic value that must be negotiated before the characters can get their hands on each other.

While the book proceeds in a brisk and brilliant succession of encounters, alternately comic and dramatic – Mr Collins's proposal, Lizzie's response, her father's reaction, Jane and Mr Bingley, Jane's illness, Mrs Bennet's hysterics, Wickham's lies to Lizzie about Darcy, Lydia and Mr Wickham – there's one apparently false note: Darcy. Has a supposedly romantic hero ever seemed less agreeable, less attractive or less charming? At a dance, he tells Bingley, in everyone's hearing, that it would be a punishment for him to dance with any of the ladies present. What, Bingham asks, about Lizzie Bennet? Darcy regards our lively, clever, witty heroine and says, "She is tolerable, but not handsome enough to tempt me."

This is not 'pride'. It's rudeness, bad manners, the words of (let us not mince words here) a stuck-up fool. We know he is "well-bred" – can breeding make one so socially maladroit? He is hopeless at conversation. He is rude to Miss Bingley. He is awkwardly icy with Lizzie. Even when he finally proposes to her, he's unforgiveably rude about her mother's vulgarity, her own "inferiority" and how degrading it would be for him to marry her. He admits, without apology, trying to derail the romance between Lizzie's sister Jane and Bingley. The reader may be forgiven for wondering when any recognisably heroic virtues will appear.

Yet in the second half of the book, he changes beyond recognition. He's discovered at his mansion, Pemberley, being charming, attentive and kind. We hear about his man-of-action heroics in persuading Wickham to marry Lydia. What has brought about this transformation?

Andrew Davies, who adapted the book for BBC TV in 1995, offered, at the time, a rather shocking analysis of the urgency of Darcy's feelings. From the start, says Davies, Darcy has been wrestling with a strong sexual attraction to Lizzie, which has upset all his notions about marriage. Must he really trade his fortune, his house and his future just for the pleasure of parting the socially inferior Lizzie from her foundation garments? It's when he decides the deal is worth it, that he becomes heroic.

Davies identifies the turning point: "There's quite a well-known scene when Elizabeth, because she's worried that her sister is ill, walks and runs across muddy fields to Netherfield. She happens to bump into Mr Darcy just as he's coming out of the house, and he finds that he responds very well to her looks. So I wrote in a stage direction: 'Darcy is surprised to find that he has an instant erection'. I felt obliged to add, 'I don't mean we need to focus on his trousers, just that it's what should be going through the actor's mind'. Darcy's obviously turned on by this heart-throbbing, muddy, warm girl."

The book's core, however, is Lizzie Bennet, so sprightly, so confident, so morally centred, such a shrewd judge of character. The arc of the narrative sees her learning to back her judgements, to make up her mind independently of others – and to learn a lesson in how the world works. It's hard for her to accept that Darcy is right about her family's low status, and the social unsuitability of their union. But in the book's climactic scene, when Lady Catherine De Bourgh explains to Lizzie why she should not presume to marry Darcy (her nephew), pride flashes a sudden fin in Lizzie's heart and she does just that.

Readers everywhere have cheered at this happy outcome. Successive generations have closed the book regretfully, sorry to leave the chattering, fretful, quarrelsome, scheming, romancing community of the novel behind. And some have pondered the paradox that such a legendarily romantic work should be centrally concerned with money, wills, land and great estates. Among them was the poet WH Auden who, in his "Letter to Lord Byron" (1937) wrote:

"You could not shock her more than she shocks me,

Beside her, Joyce seems innocent as grass.

It makes me most uncomfortable to see

An English spinster of the middle class

Describe the amorous effects of 'brass',

Reveal so frankly and with such sobriety

The economic basis of society."