Over the past few weeks, punters in China underwent a near-death experience when their country’s two stock exchanges entered freefall. The rapidly inflating bubble that had driven share prices to dizzying heights had suddenly burst. By this spring, the stock markets in Shanghai, with 831 listed companies, and Shenzhen, with 1,700, boasted a combined market capitalisation of $9.5tn, which made them – along with the much older Hong Kong exchange – the second-largest financial market in the world.

After languishing for the past four years, these two Chinese stock markets suddenly took off last summer, becoming a cauldron of voracious buying, selling and spectacular profit-taking. Shares of newly listed companies soared thousands of percentage points within months of their initial public offerings, driven upward by a new and growing cadre of relatively unsophisticated private investors that included tens of millions of ordinary workers, farmers, housewives and pensioners. According to one widely cited survey of these new investors, 67% of them have less than a high-school education.

The fact that Chinese stocks were climbing ever higher while the Chinese economy was cooling should have been an unmistakable warning of a bubble, but it caused surprisingly little concern. (Another reason to worry might have been the disparity in prices between so-called “A-shares”, which can only be purchased by investors inside China to keep the domestic market shielded from outside foreign manipulation, and stakes in the same companies available to foreign investors through the Hong Kong exchange, known as “H-Shares”. This disparity suggested Chinese investors were bidding up prices well beyond any reasonable approximation of their value.) In fact, drawn by the casino-like profits to be made in the boom, more and more small investors flocked to the thousands of brokerage houses that are now proliferating in every Chinese city in order to buy and sell while staring up at flickering electronic data boards charting the rise and fall of equity prices.

With markets rising in straight lines on graphs plotting their progress – the Shanghai exchange had shot up some 135%, and the Shenzhen exchange had gone even higher at 150% in less than a year – stocks had begun to seem like a sure bet for Chinese investors with fevered dreams of quick wealth. They promised a much higher rate of return than traditional low-interest bank savings accounts, which have paltry annual yields of barely 2%. At the time of the crash, 9% of Chinese households – some report the figure as high as 200 million people – had bought into the booming market. Steadily rising prices seemed to be delivering on both Deng Xiaoping’s promise of “a relatively well-off society” (xiaokang shehui) and the current president Xi Jinping’s rhetoric of a full-blown “Chinese dream” (zhongguo meng) – a fuzzy notion that promises wealth, wellbeing, and power to individuals and the nation as a whole.

China had already experienced a dangerous bubble in its residential housing market, but in that case the government had succeeded in engineering a relatively soft landing by raising interest rates, limiting the number of residences one owner could buy in such cities as Beijing and Shanghai and levying a new tax. Accordingly, it was all too easy for small investors to assume that the bull market was implicitly backed by a kind of unwritten government guarantee – that the good times were only beginning to roll, and the state would step in to take care of things before the bottom fell out. In fact, the government itself had become bedazzled by the seemingly invincible rise in stock prices. Instead of dedicating its energy to regulating the markets, the Chinese Communist party began to see an unprecedented opportunity in further inflating the bubble – a chance to sell equity stakes in dangerously debt-burdened state enterprises and help clean up some very messy balance sheets. If the planting of two stock markets on soil long ploughed by Maoist sloganeering about “capitalist roaders” was a mild surprise, it was mind-bending to witness the party embrace the bull market so ardently that even its official voice, the People’s Daily, began to flog stocks as a golden risk-free opportunity.

When the Shanghai index reached a new pinnacle of 4,000 in April, a column in the People’s Daily effused that this “was only the start of a bull market”. “What’s a bubble?” it asked insouciantly. “Tulips and bitcoins are bubbles … But if A-shares are seen as the bearer of the Chinese dream, then they contain massive investment opportunities.”

With the party cheerleading the market’s inexorable rise, it became even easier to imagine that the government was, in effect, declaring an informal debt obligation on the stock market’s future – essentially covering any bets with its own considerable assets. How could you lose? And so the bubble grew and grew: price-to-earnings ratios for Chinese stocks averaged an astounding 70-to-1, against a worldwide average of 18.5 to 1; the value of the A-shares inside China grew to be nearly double the equivalent shares of the same companies on the Hong Kong exchange. Ordinary Chinese people had become so intoxicated by bull-market euphoria that stories began to proliferate about people leaving their jobs, and even their families, to become day traders, often using funds borrowed from high-interest rate “shadow banks” or loans taken out against their homes. (By last weekend, margin borrowing on both exchanges had surpassed $320bn, representing almost 10% of the total market capitalisation of all stocks being traded.)

Ordinary Chinese had become so intoxicated by bull-market euphoria that stories spread about people leaving their jobs

Of course, some observers could see the inherent weakness in this seemingly irrepressible market, but with so much money to be made, it was easy to ignore or disparage warnings of anomalies and irregularities. Few wanted to ruin the heady ride with naysaying and pessimism, particularly as some 7% of households had invested 15-20% of all their assets in shares. When the World Bank released its China Economic Update Report this spring, noting that the state had gone beyond its role as a regulator and guarantor for financial systems and engaged in more active interference, its officials met with strong opposition from the Chinese government.

“Instead of promoting the foundations for sound financial development,” the World Bank’s report cautioned, “the state has interfered extensively and directly in allocating resources through administrative and price controls, guarantees, credit guidelines, pervasive ownership of financial institutions and regulatory policies.”

It was a prescient warning, given what followed. But almost immediately after the report appeared, the chapter containing these cautions suddenly vanished: unspecified Chinese officials had taken umbrage at such direct criticism, and forced the World Bank to redact the offending portion of its analysis. Chinese officialdom has often had difficulty accepting criticism – especially from outside experts in a public setting – and whether this critical shortcoming is a consequence of traditional mores or a Leninist political culture is hard to determine. However, such sensitivities have frequently prevented Chinese officials from identifying and fixing problems before they erupt into crises. In the case of China’s stock market bubble, almost no one wanted to listen to voices of caution – and the whole country was to pay a bitter price for avoiding reality. For what was at stake was not just the integrity of China’s financial system, but the question of the Chinese people’s ongoing confidence in their government. This would be critical as the country continued, in Deng Xiaoping’s words, “to cross the river by feeling its way over the stones” (mozhe shitou guohe). Indeed, there is no nation of significance in the world today that is in the process of undergoing a more challenging transition: from Maoist revolution to stakeholder in the modern, marketised global world.

A Chinese investor looks at falling share prices at a brokerage house in Hangzhou city in east China on 7 July. Photograph: Reuters/China Daily

When the Shanghai stock exchange composite index plunged from its high of 5,166 to just about 3,700 over the course of a few short weeks in June and early July – wiping out some $3tn worth of market value – its precipitous fall stunned the nation. The collective sense of shock owed something to the fact that over the past 25 years, China had not encountered many economic crises that officials found impossible to handle through state intervention. So, when this one began, the natural reaction was to attempt to “manage” the collapsing share prices through more state intervention, beginning with a surprise interest rate cut on 27 June. As if the stock markets were still just cogs in a socialist command economy, the government then ordered up a whole menu of non-market fixes: first, a moratorium on all new public offerings; then, on 8 July, a six-month prohibition on share sales by company directors or any listed shareholder owning a 5% stake in that company. By this time, more than half of the listed companies on the two Chinese exchanges had suspended trading of their own stocks; state-owned enterprises (SOEs) were instructed not to sell shares.

At the same time, the China Securities Regulatory Commission created a “market stabilisation fund” of $19.3bn and ordered brokerage houses to start buying into the market themselves. The commission also relaxed rules on margin trading so that government pension funds and other SOEs could also answer the party’s call to prop up the market themselves by buying shares; it lowered securities trading fees; and even promised to purchase shares through its own proprietary accounts, and to continue doing so for as long as the composite index for both exchanges remained below the new target of 4,500.

To reassure “the people” that the government had not turned bearish on the foundering markets, the People’s Daily again rhapsodised about the glories of investment over “the long term”: “It is after storms that we encounter rainbows,” it wrote. “Looking back at the development of China’s capital market, we realise that the road to development has not always been smooth, but has instead been a twisting one with ups and downs. But it is in each lesson learned that the market has matured … So participants in the market should earnestly reflect, collectively sum up their experiences, and then work together to achieve a capital market that is stable and can continue to develop in a healthy manner over the long term.”

But such florid rhetoric was too little and too late to soothe the agitated investors who had just lost their shirts. Not surprisingly, the unofficial press and the internet had erupted with the sentiments of people furious about both their financial losses and what they regarded as the Communist party’s poor leadership.

One young investor in Manchuria blamed “the state’s inadequate regulation”, and pointed to the government’s role in inflating the bubble as a reason for it to forcefully intervene to prop up prices. “The bull market was itself a policy-driven one, so only major policies can save it,” he told the New York Times.

Another angry investor – who claimed to have lost most of his savings – posted a complaint that quickly went viral: “This was a stock wipeout that thoroughly damaged middle-classed assets from a decade of striving,” he wrote. “For us the ‘Chinese dream’ is just a dream.”

This was a particularly pointed attack on Xi’s own roseate notion of the “Chinese dream” – which the president has relied upon to inspire his people forward on what he calls “the road to rejuvenation”, and to galvanise their nationalism against the various “hostile foreign forces” (jingwai didui shili) that he regards as a threat to China’s continuing rise.

Deng Xiaoping in 1986. Photograph: Reuters/Corbis

Even China’s feared Public Security Bureau was summoned to meet the challenge of the plunging stock markets. Its charge was to investigate, and possibly arrest, what the New China News Agency called “malicious” short sellers (which in China is not an illegal practice) – a group of malefactors said by some Chinese media outlets to include the American financier George Soros.

The extent of the party’s belated panic may have been best captured by the absurd instruction given by party propaganda officials to students at Tsinghua University School of Economics and Management, who were told to loudly chant, at the start of their graduation ceremony, “Revive the A-shares! Benefit the people!” (After news of this effort went viral, the chant was changed at the last minute to something more anodyne.)

In 1989, four days after the massacre of protesters at Tiananmen Square, a gloomy Deng Xiaoping told his military commanders, “There was this storm that was bound to happen sooner or later.” He was doubtless referring to the way he had opened up Chinese society over the preceding decade to many democratic ideas, which helped precipitate the monumental demonstrations in Beijing that spring. Were Deng still alive to watch the Chinese stock exchanges that he himself had initiated crash some 25 years later, he might have said the same thing. For the rise, and fall, of these markets was a delayed but equally inevitable outcome of another one of his very unlikely and risky experiments.

Upon returning to power in 1978 after Mao’s death, Deng feared that China would fall further behind the rest of the world if it did not quickly initiate radical reforms. Proclaiming a bold new programme of “reform and opening up” (gaige kaifang), he called on the Chinese to begin borrowing from other economic and political systems. “We mustn’t fear to adopt the advanced management methods applied in capitalist countries,” he said.

It was a shocking exhortation from a communist leader who had spent his entire life steeped in Maoist revolution and class struggle. But it was only the first of a string of surprising public utterances and policies during these early pioneering years in the 1980s. Part of Deng’s broad regimen of reform involved testing the transplantation of indelibly capitalist ideas and institutions into the host body of China’s hitherto completely Marxist-Leninist system.

One of his boldest experiments began in the late 1980s, when, like a mad scientist playing with the creation of new hybrid species, he called for the creation of two capitalist-style stock exchanges in China. The first was to be established in Shanghai, a city that had been the wellspring of the most virulently leftist form of violent Maoist class struggle during the Cultural Revolution. The second was to be set up in Shenzhen, then a brand new city-in-the-making that was itself another one of Deng’s bold experiments. It involved the creation of four new coastal special economic zones, intended to promote Chinese development by allowing more unfettered interaction with the outside world.

Shanghai Stock Exchange. Photograph: Eugene Hoshiko/AP

Observers at the time – including myself, then covering these counterintuitive changes in China for the New Yorker – were left scratching our heads in puzzlement. How were these capitalist organs ever going to be transplanted into the living body of Chinese socialism, much less take root and thrive?

Deng, who quaintly dubbed all these experiments “socialism with Chinese characteristics” (zhongguo you teside shehui zhuyi), seemed unfazed by all the contradictions inherent in these plans. But, as someone who had first come to China during the Cultural Revolution, I found it difficult not to be sceptical about such projects succeeding – given the clash between what then appeared to be two wildly irreconcilable ideological positions. When the Shanghai exchange opened its first proper trading floor just across the Garden Bridge in what, before 1949, had been the grand old colonial Astor Hotel, it was a mind-bending experience to gaze out over the traders’ desks to the electronic index boards (some of the first I had seen in China!) where the share prices of the few score listed companies were displayed. It was hard not to wonder how Chairman Mao’s vaunted Chinese communist revolution had ever come to this. How had such a reviled capitalist institution fallen into such an apostate land?

However, as Deng’s new amalgam economy began to gather momentum, private businesses did begin to proliferate and the economy did begin to take off. Soon, the sceptics were left to wonder whether Deng might not have stumbled on a new hybrid economic model after all – one that was not only workable, but very dynamic. If so, he had done a masterful end-run around all the old verities of our own western economic development theory, systems and experience.

But as bold, inventive and successful as Deng Xiaoping’s new economy came to seem over the ensuing years, the implanting of such market-oriented entities as stock exchanges into what was still largely an unreconstructed one-party, centrally planned state economy created a situation that seemed destined – as Deng had lamented in 1989 – to trigger a storm sooner or later. In the world of economics, there are few institutions more dependent than stock exchanges on an ability to respond in an unfettered way to market forces. After all, in the world of politics, however, there are few systems more dedicated to maintaining central control and empowering state intervention than that of the People’s Republic of China. In short, it seemed highly unlikely these opposite tendencies could coexist in happy synergy for ever: what we have seen playing out in China over the past few weeks, then, was a kind of delayed autoimmune reaction to having such an alien presence transplanted within it. The host might survive, but the organ was going to have to adapt to be accepted.

By the end of last week, Chinese markets had started to respond to the government’s many interventions, and a slight rebound began. But although they were still up 82% over a year ago, they had fallen 28% from their high in June – and this week, they began dipping once again. Furthermore, some half of all listed companies – representing 40% of the market’s total value – had suspended trading, creating a gross market distortion, augmented by the fact most “buyers” in the market were now government-funded surrogates ordered to do so, not value-conscious investors.

It is hard to imagine that all this behind-the-scenes manipulation will not dent the confidence of investors in the future: the already tenuous connection between share prices and actual corporate value will now be even more uncertain, when the government, in effect, has its thumb on the scales. Added to that fear is the real possibility that shareholders who re-enter the market may find themselves in the future holding untradable and therefore illiquid shares if the government again decides to freeze market operations in response to a sharp decline. Almost immediately after the crash, the overseas investors who had bought into Chinese companies through the Hong Kong exchange began pulling their capital out of Chinese stocks – the most prolonged period of net outflow since the programme of trading via Hong Kong began last November.

So it is still far too early to speak of China’s stock markets as having “stablised” or “returned to normalcy”. (To begin with, many company shares are still not trading.) By making the risky choice to move in and prop up share prices as they fell, party leaders effectively took ownership of markets whose proper functioning requires them to remain independent. Evidently, they felt that their credibility as China’s grand air-traffic controllers was being put at risk – that they would lose credibility, respect and even face if they did not confront the plunging market head on and at least give the appearance of being in control.

Here it must be said that, because China is a closed society and the negotiations among its leadership are so opaque, we on the outside rarely know who really makes decisions about important issues – or by what process they are made; all we can see are the results that follow. So when we speak of the wishes of “party leaders”, or even of “President Xi”, we are referring to a world that we can only see through a glass darkly.

The party felt its credibility as China’s grand air-traffic controller was at risk – that it would lose credibility

But in this case, the consequences of the decision are clear. By acting so intrusively, party leaders have left themselves subject to what Colin Powell memorably called the “Pottery Barn rule” – if you break it, you own it. Suddenly, China’s stock exchanges have become wards of the Chinese Communist party – and their fate hardly bodes well for Xi’s declaration that the nation’s economic salvation will lie in allowing market forces to play a greater role in the allocation of resources.

There is an ancient Chinese aphorism: “Sleeping in the same bed, but dreaming different dreams” (Tongchuang yimeng). By initiating so many innovative reforms back in the 1980s, Deng Xiaoping not only created some strange bedfellows, but bequeathed a complex legacy of interbred institutions to later generations. By introducing so many conflicting ideas and institutions from different systems into the heart of what was still the Chinese communist revolution, Deng Xiaoping became the progenitor of what ended up being a virtual counter-revolution against Maoism. And while his “reform and opening up” did infuse his country with significant new dynamism, it also put a series of troublesome institutional and ideological contradictions at the centre of China’s whole post-Mao landscape. Stock markets were only one of the most obvious and graphic examples of these contradictions.

When the stock bubble – which had so inflated the listed value of shares that a natural correction was inevitable – finally burst, China did appear to be undergoing a kind of organ failure. In the view of Beijing’s attending neo-Maoist physicians, this failure called for emergency life support. Ever worried about any perturbation in the field that might undermine social stability, China’s “socialist” apparat did what it knows how to do best when alarmed. Instead of standing by and waiting while Adam Smith’s “invisible hand” restabilised the markets – albeit at a considerably lower valuation – Xi and his leadership sprung into action. Having evidently decided that this precipitous correction was a dire threat not just to China’s economic health, but also to social and political stability, they sought to levitate share prices back up again through whatever forms of state intervention and manipulation they could muster.

A Chinese worker walks past a billboard along a construction site in Shanghai. Photograph: Philippe Lopez/AFP/Getty Images

The party might have been excused if it had simply eschewed responsibility for what was happening. After all, markets have a logic of their own that makes them both rise and fall according to their own forces. But, instead of simply saying, “Not our problem”, it launched a massive socialist-style rescue campaign, thereby making the party responsible for everything that happened thereafter.

Why did the party allow itself to become stuck in this quicksand? Leaders evidently felt themselves threatened not only by the collapsing share prices, but by what they also feared would be perceived as an erosion of their own credibility. What they seem to have concluded was at stake was their ability to continue projecting an image of omnipotence – the appearance, at least, of being strong enough to continue guiding and controlling “all under heaven” (tianxia).

This is an ancient notion, dating back to imperial times, when an emperor’s reign was believed to be legitimised by a so-called “mandate of heaven,” (tianming), that conferred the right on a sovereign to rule. Any untoward sign of heaven’s disfavour, it was believed, would be manifested through such things as earthquakes, rebellions, droughts or other disturbances in the usual order of things. And since such events were invariably viewed as ominous end-of-dynasty symbols, emperors were always strongly allergic to them.

Of course, emperors of old never had to wrangle modern stock markets. But today, a good market crash fits right into these ancient – and still deeply embedded – ideas in the popular mind. Indeed, in the modern context, it is not hard to see how a crashed financial market might be viewed as a powerful suggestion that party leaders are losing heaven’s favour and their own legitimacy, and, worse, that a new dynastic cycle may be in the offing. And, of course, the last people the party wished to alienate were members of its burgeoning middle class, whose quiescence has long depended on the economy moving onward and upward without major interruption.

Since such residual traditional logic remains deep in the bloodstream of the Chinese people, it most certainly played a role in goading party leaders into attempting to bring these two insubordinate “capitalist” markets to heel as quickly as possible to provide proof that they still held the right to rule. Of course, when such modern markets are rising, they ipso facto help prove that “the mandate” is still conferring legitimacy on China’s rulers, and so can be left unattended. But when they begin to founder, the ensuing dysfunction creates the suggestion that the mandate is weakening. Such a threat begs urgent intervention.

The obvious contradiction between a largely self-regulating financial market and a highly controlled and centralised economy is a graphic representation of China’s divided modern-day self. What characterises China today is that it is in the middle of a process of uncertain change in many ways. Its once purely socialist command economy is now partially socialist and partially capitalist, and it is this collision that helped trigger the drama of clashing systems and values we have been watching play out over the past few weeks.

President Xi, like his predecessor Hu Jintao, speaks often about the Confucian virtues of harmony (hexie) and stability (wending). Perhaps this is because every time he and the six other members of the ruling politburo standing committee turn around, they run into disaccord and instability. And many of the problems they now confront grow out of the fact that, for the past century, China has been in the process of tectonic self-reinvention, seeking to recreate not only a new political system, economic structure and value system, but a whole new national identity. This long, painful, and complicated process began with the collapse of China’s ancient imperial system and its sustaining Confucian ideology in 1912. Thereafter, China was left with the daunting task of filling the remaining vacuum – of recreating itself from the ground up. In serial fashion it experimented with Sun Yat-sen’s Republicanism, Chiang Kai-shek’s east-west syncretism, Mao Zedong’s sinified communism, Deng Xiaoping’s hybrid capitalist-socialism, Jiang Zemin and Hu Jintao’s consensus authoritarianism; and now … now what? What will the essence of Xi Jinping’s reinvented China be? What exactly is his vision of financial markets in a vibrant new economy? One that does the party’s bidding or one that responds independently to market signals and market value, as other global capital markets do? And what is his larger vision for China’s economy, political system, rule of law, civil society, and system of core values? What, in effect, is his model?

Although we can see a few shadowy outlines of answers emerging as China’s reform odyssey continues, we still do not really know exactly where Xi intends to take the nation. To look into his “Chinese dream” is to see an aspiration for a country that is wealthier, more powerful and better respected. If you look at Xi’s domestic policies it is possible to see an ominously Mao-tinged autocrat whose answer to most problems seems to be more discipline, controls and toughness. But there is little else. And so, because China will almost certainly remain caught between transitions for some time to come, the resolution of crises such as stock market crashes will remain an uncertain and parlous business. The Maoist toolbox into which Xi now seems to reach with increasing frequency when problems occur provides him with few suitable tools for handling many of the complexities of 21st-century economic markets.

China’s future remains unclear, because so much of its agenda is still a work in progress. To have a capitalist stock market being played like a casino by tens of millions of freebooting speculators right in the middle of a society still purporting to be socialist and run by a communist party with a deep affinity for rigid, Leninist, interventionist controls speaks to the contradictory nature of the modern Chinese dilemma. It is a living embodiment of what Mao Zedong liked to refer to as an “antagonistic contradiction” (diwo maodun) – an inconsistency that can only be resolved through struggle, even violence.

Alas, such an unenviable predicament is hardly limited to China’s stock markets. In certain telling ways the response of the nation’s leaders to the recent market crash is emblematic of a much larger dilemma – one that sits right at the heart of China’s uneasy fusion of communism and free-market economics, a system with little precedent and no operating manual. In the years to come, with many opposing principles and forces in an unresolved state of contention, it is unlikely, as China’s great experiment in self-recreation goes forward, that its hallmarks will be “harmony” and “stability”.

• Follow the Guardian long read on Twitter: @gdnlongread