When I was a graduate student in Oxford many years ago I shared a house with a brilliant German sinologist who used to push translations my way, stroking his beard with a teasing smile: “Try this – you’ll really enjoy it.” Many visitors popped into our terraced house on Abingdon Road, and one night around the kitchen table I met a fascinating character, rangy with white hair and beard, and a twinkly eye. His name was David Hawkes.

A gifted linguist, he had directed Japanese codebreakers in his early 20s, during the second world war. As a student at Peking University, he had been in Tiananmen Square in 1949 when Mao proclaimed the People’s Republic of China. Later, as a teacher, he had done a wonderful translation of the Songs of the South, part of a poetic tradition earlier than anything that has survived in the west. Then he became professor of Chinese in Oxford, but, as he put it, “I resigned in order to devote my time to translating a Chinese novel … well, the Chinese novel”.

The book was Dream of the Red Chamber, also known as The Story of the Stone, written by Cao Xueqin. The critic Anthony West called it “one of the great novels of world literature … to the Chinese as Proust is to the French or Karamazov to the Russians”.

An illustration from the book

Hawkes eventually completed his great endeavour with the help of his son-in-law John Minford, who finished the last two volumes of the five, which were published by Penguin between 1973 and 1986. Hawkes’s translation was greeted as an introduction to “a masterpiece”, a “work of genius”, a “candidate for the Book of the Millennium”. When Chinese premier Wen Jiabao was given a copy of Shakespeare during his state visit to the UK, the new Chinese ambassador Fu Ying gave the queen the Hawkes translation.

The novel is an 18th-century saga, the tale of a noble family that falls from grace. It is full of incredible detail of the social, cultural and spiritual life of the time. Chairman Mao claimed to have read it five times – and thought everyone else should too. Today, everyone in China knows it, partly due to the much-loved 1987 TV version, which had the impact of the Colin Firth/Jennifer Ehle Pride and Prejudice in the UK.

Dream of the Red Chamber was written in the 1750s “by a great artist with his very lifeblood”, said Hawkes. Cao’s story mirrors the tale of his own family. His grandfather, Cao Yin, was an imperial bondservant, an important functionary in the south, who enjoyed high favour with the emperor Kangxi. But after Kangxi died, his son began a purge of corruption and incompetence, and the family were ruined. They lost their mansion in Nanjing and moved to a modest house among the alleys of Beijing, south-east of the Forbidden City.

So Yin’s grandson grew up in straitened circumstances, a brilliant but watchful boy, wary of all power, and never forgetting his grandad’s saying about the fickleness of fortune: “When the tree falls, the monkeys will be scattered.” He was good with the brush, both with paint and with words: but he had no aptitude for university, so he found himself down and out in his 30s, selling his paintings and working as a private teacher (he was eventually sacked for getting a maidservant pregnant.) By the end he was sleeping in barns and working in wine shops; he clearly drank too much.

A performance of Dream of the Red Chamber at the Cairo Opera House in 2010.

The book was written in dribs and drabs: each new chapter circulated among family and friends, often in exchange for a meal and a pitcher of wine. He died in 1763, heartbroken it is said, by the death of his only son. Dream of the Red Chamber was finally published in print in 1791, but the text is still surrounded by controversy. There is a story that it had been censored because eminent people it satirised had been too thinly disguised. It is also debated whether the text we have is all his. Different endings survive, with a writer called Gao E claiming to have published the complete version according to the author’s wishes. Today, “redology” (Red Chamber obsessives are known as “redologists”) is a massive and still expanding field in China, with conferences, annual journals and a torrent of publications. Manuscripts still turn up. Mysteries remain unsolved.

Hawkes’s version gives us the first 80 chapters by Cao and the last 40 redacted by Gao E, who it seems fair to assume was using an unfinished draft and clearly knew what the author had planned. It is a different kind of novel from earlier Chinese classics such as Romance of the Three Kingdoms, The Water Margin and Monkey, the latter a vast sprawling narrative, surreal and poignant, full of songs and poems. The female characters are especially strong. As Cao himself said: “Having made an utter failure of my life, one day I found myself in the midst of my poverty and wretchedness, thinking about the female companions of my youth. As I went over them in my mind’s eye one by one it suddenly came over me that those slips of girls – which is all they were then – were in every way, both morally and intellectually, superior to the ‘grave and moustachioed signor’ I am now supposed to have become. The realisation brought with it an overpowering sense of shame … And I resolved then, however unsightly my shortcomings might be, I must not, for the sake of keeping them hid, allow those wonderful girls to pass into oblivion without a memorial.”

The book as it stands in the Penguin version runs to 2,500 pages – twice as long as War and Peace. Hard going at first because of the myriad characters (there are 40 main ones) and their (to a non-Chinese eye) difficult names. But once you are into it, it is a book into which the reader can completely immerse herself; it is like nothing else in all of literature. Having just watched his thrilling adaptation of War and Peace, all I can say is: “Over to you, Andrew Davies.”

• The Story of China is on BBC2 on Thursday.