When I moved to Beijing in August 2003, I believed I had the best job in the world: working for my favourite newspaper in the biggest nation at arguably the most dramatic phase of transformation in its history. In the past decade, it has given me a front-row seat to watch 200-odd years of industrial development playing at fast forward on a continent-wide screen with a cast of more than a billion.

That said, I am glad my daughters were young and easy to please back then or we might well have taken the first plane out of the country. As we drove from the airport to our apartment, I tried to maintain an upbeat chatter. "Look at all the kites," I said as we passed Chaoyang park, even though my heart sank at the tatty buildings, endless construction sites and stultifying haze. In my head, I asked myself: "Have I done the right thing for them?"

We had come from Japan – a democratic, comfortable, polite, hygiene-obsessed, orderly, first-world nation – to the grim-looking capital of a developing, nominally communist country that looked and sounded like a giant building site. For all my enthusiasm, my family must have felt we were taking a step backwards in lifestyle.

It required an adjustment of preconceptions. Like many newcomers, I delighted at discoveries of Chinese literature and Daoist philosophy, Beijing parks, the edgy eccentricity of Dashanzi and the glorious mix of classicism and obscenity in the Chinese language, though I never managed to master it. The mix of communist politics and capitalist economics appeared to have created a system designed to exploit people and the environment like never before. It was so unequal that Japan appeared far more socialist by comparison. And it was changing fast. As swaths of the capital were being demolished and rebuilt for the Olympics, there was an exhilarating (and sometimes disorientating) sense of mutability. Everything seemed possible.

Looking back over the stories that followed, it is hard to believe so much could be compressed into such a short span of time – the outbreaks of Sars and bird flu, the attempted assassination of the president of Taiwan, deadly unrest in Tibet, the devastating earthquake in Sichuan, murderous ethnic violence in Xinjiang, Tibetan self-immolations, as well as the huge regional stories: two tsunamis – in 2004 in the Indian Ocean and last year in the Pacific, a multiple nuclear meltdown in Fukushima, and the protracted rattling of nuclear sabres on the Korean peninsula.

One of my first tasks in 2003 was to chose a Chinese name. I opted for "Hua Zhong", partly because it sounds a little like "Watts, Jon", but mainly because the characters mean "Sincere to China" – something I was determined to be as a reporter. Nine years later, that sentiment has not faded. But at various times, I have been called a communist sympathiser, supporter of Taiwan, a stooge of the Dalai Lama.

However, my focus has been on development and its impact on individuals and the environment. In 2003, China had the world's sixth-biggest GDP. It passed France in 2004, Britain in 2006, Germany in 2009, and Japan in 2011. On current course, it will replace the US as No 1 within the next 15 years. It is already top in terms of internet population, energy consumption and the size of its car market.

A decade of dominance

The primary driver for change has been the movement of people. Over the past nine years 120 million Chinese people – almost twice the population of the UK – have moved from the countryside to the city. This mind-boggling shift has its problems, as I found in Chongqing, but for the most part, China appears to have avoided the worst of the poverty, crime and ghettoes seen in other rapidly urbanising countries.

Yet it also seems more brittle, perhaps because of the other big economic engine: infrastructure investment. In this period, China has been re-wired and re-plumbed. There has been an extraordinary expansion of power, transport and communication networks that have linked the nation like never before: west-east gas pipelines, south-north water diversion, hundreds of airports and a massive new electricity grid linking wind and solar power plants in the deserts to power-hungry consumers in cities and industrial plants.

This has been a decade of cement and steel, a time when economic development has pushed into the most remote corners of China with a series of prestige projects: the world's highest railway, the biggest dam, the longest bridge, putting a man into space, the most ambitious hydroengineering project in human history and, of course, hosting – and dominating – the Olympics for the first time.

It has been a privilege to watch this redistribution of wealth and power from the developed to the developing world. On a global level, such a shift will require nothing less than a grand accommodation – or a violent conflict. Beijing appears to be preparing for both. Other news during this period showed a hardening of China's military muscle: a breakthrough satellite-killing missile test, the launch of a first aircraft carrier, the development of a new stealth fighter and a steady increase in military spending.

Criticism has rarely been appreciated. All too often, there have been flare-ups of anti-foreign media hostility. Some of my colleagues in other media organisations have received death threats. I never expected China to be an easy place to work as a journalist. For political and cultural reasons, there is a huge difference in expectations of the media. For historical and geo-strategic reasons, there is a lingering distrust of foreign reporters.

Run-ins with the police, local authorities or thugs are depressingly common. I have been detained five times, turned back six times at roadblocks (including during several efforts to visit Tibetan areas) and physically manhandled on a couple of occasions. Members of state security have sometimes followed interviewees and invited my assistants "out for tea", to question them on who I was meeting and where I planned to visit. Censors have shut down a partner website that translated Guardian articles into Mandarin. Police have twice seized my journalist credentials, most recently on this year's World Press Freedom Day after I tried to interview the blind human-rights activist Chen Guangcheng in hospital. When that happened, I debated with another British newspaper reporter who was in the same position about whether to report on the confiscation. He argued that it was against his principles for journalists to become part of the story. I used to believe the same, but after nine years in China, I have seen how coverage is influenced by a lack of access, intimidation of sources and official harassment. I now believe reporters are doing a disservice to their readers if they fail to reveal these limitations on their ability to gather information.

Yes, there is often negative coverage and yes, many of the positive developments in China are underemphasised. But I don't think it does the country's international image any favours to clumsily choke access to what is happening on the ground.

Treated like a spy, I have sometimes had to behave like one. At various times, I've concealed myself under blankets in a car, hidden in a toilet, waited until dark in a safe house and met sources in the middle of the night to avoid detection.

At other times, it is Chinese journalists and officials who pull the screen of secrecy aside. Take the foot-and-mouth outbreak on the outskirts of Beijing in 2005. I was first alerted to this by a Chinese reporter, who was frustrated that the propaganda department had ordered the domestic media not to run the story.

Foreign ministry officials often tell me China is becoming more open and, indeed, there have been steps in that direction. But restrictions create fertile ground for rumour-mongering. One of the biggest changes in this period has been the spread of ideas through mobile phones and social networks. The 513 million netizens in China (up from 68 million in 2003) have incomparably greater access to information than any previous generation and huge numbers now speak out in ways that might have got them threatened or detained in 2003. Microblogs are perhaps nowhere more influential than in China because there is so little trust of the communist-controlled official media.

It has been fun watching netizens create an ingenious new language to evade restrictions. In this anti-authoritarian world, the heroes are the "grass mud horses" (which, in Chinese, sounds the same as a core banned phrase: "Fuck your mother!") while the villains are the river crabs (which is pronounced like "harmony" – the favourite excuse of the authorities when they crack down on dissent). But ultimately, a journalist wants to see things for him or herself. I will never forget the epic road trips – across the Tibetan plateau, along the silk road, through the Three Gorges and most memorably from Shangri-la to Xanadu. Along the way, I met remarkable people with extraordinary stories. True to the oft-heard criticism of the foreign media, many were from the "dark side": a young man in Shaoguan who confessed – as the shadows lengthened on the building site where we had our interview – to killing Uighur co-workers at his toy factory because of a rumour they had raped Han women; a gynaecologist in Yunnan who argued with great conviction that it had once been necessary to tie pregnant women up to carry out abortions; the young boy who found the body of his dead grandmother who killed herself a year after his father – an illegal migrant – phoned her to say he was about to drown in what became known as the Morecambe Bay disaster.

Other stories literally turned up on my doorstep – such as the petitioner who arrived at my office a few weeks before I left. We had never met, but it was easy to identify Yang Zhong, who stood out a mile with his country boots, green overalls and bags crammed full of injustice. The look was all too familiar. I have lost count of the number of petitioners who have asked the Guardian to investigate land thefts, corruption cases, industrial accidents, rapes, murders and other alleged abuses of power.

Yang had come from Jinshantun village in the far northern province of Heilongjiang to accuse a local forestry chief – Wang Liguan – of illegal logging in one of China's last great protected forests and for having him locked up and beaten when he dared to complain. Typically, his story, minutely recorded in reams of papers and even on a DVD, was anything but black and white.

Weak laws and strong censorship make it difficult for such people to have their cases heard in the domestic system so they turn to foreign news bureaus. There were so many grim accounts it was impossible to give them all the time and attention they deserved.

Heroism and brutality

But there were also stories of success, heroism and inspiration as a nation embraced new wealth and battled for new ideas: the business empires built by enlightened philanthropists such as Yin Mingshan of Lifan auto, the internet fortunes accrued by entrepreneurs such as Jack Ma of Alibaba and Robin Li of Baidu.

Compared with nine years ago, people in China have more freedom to shop, to travel and to express their views on the internet. The Communist party tolerates a degree of criticism,but step over the invisible line of what is acceptable and the consequences are brutal. In my first years in China, I interviewed several outspoken opponents – Liu Xiaobo, Gao Zhisheng, Hu Jia and Teng Biao. I was impressed back then that they were at liberty to speak out. It seemed like the act of a confident government. But all of them have subsequently been locked up and, in at least two cases, tortured.

The blame for that surely lies with the authorities. But I have sometimes felt pangs of guilt. I first interviewed Ai Weiwei in the summer of 2007 for an Olympic preview. He was one of the creators of the "bird's nest" stadium and I was expecting him to tell me how proud he would be when it was unveiled at the opening ceremony. Instead, he told me he would not attend in protest at the "disgusting" political conditions in the one-party state and then launched into a withering assault on propaganda. It was the first time he had expressed such views to the foreign media – a great scoop, but also one fraught with risk. At the end of the interview, I cautioned the artist: "Are you sure you want to say this? It could get you into a great deal of trouble with the authorities."

"Absolutely," he replied. "I only wish I could say it more clearly."

Despite that confirmation and the similarly critical comments he subsequently made to other media organisations, I felt partly responsible when Ai was detained last year.

Whether the repression is getting better or worse has been a constant question with few clear answers. My feeling is that China has become a less tolerant country since 2008.

That was a coming of age of sorts, when China stopped seeming like a work in progress and started looking and behaving like a superpower. On the Beijing skyline, the scaffolding and cranes had been replaced by stunning architectural wonders. The ever-present sentiments of victim-hood and nationalism found powerful outlets in the Tibetan uprising, torch relay protests and the Sichuan earthquake. Meanwhile, those who had supported moves towards a more open, liberal, internationalist China saw the value of their political stock plunge almost as fast as the Dow Jones index in the global financial crisis. With the western model apparently shattered, many in China understandably felt less inclined than ever to listen to outside advice.

In the four years since, China has become a more modern and connected nation, but – despite the official hubris – it also seems more anxious that the uprisings in the Middle East and North Africa may spread. The government now spends more on internal security than defence of its borders – a sign that it is more frightened of its own people than any external threat.

Little wonder. This has been an era of protest in China. The government stopped releasing figures a few years ago, but academics with access to internal documents say there are tens of thousands of demonstrations each year. The reasons are manifold – land grabs, ethnic unrest, factory layoffs, corruption cases and territorial disputes. But I have come to believe the fundamental cause is ecological stress: foul air, filthy water, growing pressure on the soil and an ever more desperate quest for resources that is pushing development into remote mountains, deserts and forests that were a last hold-out for bio and ethnic diversity.

This is not primarily China's fault. It is a historical, global trend. China is merely roaring along the same unsustainable path set by the developed world, but on a bigger scale, a faster speed and at a period in human history when there is much less ecological room for manoeuvre. The wealthy portion of the world has been exporting environmental stress for centuries. Outsourcing energy-intensive industries and resource extraction have put many problems out of sight and out of mind for western consumers. But they cannot be ignored in China.

The worst problems are found in the countryside: "cancer villages", toxic spills, pitched battles to block a toxic chemical factory, health hazards from air pollution and water and the rapid depletion of aquifers under the north China plain – the country's bread-basket.

The implications are global. China has become the biggest greenhouse-gas emitter on the planet. This year, it will probably account for half the coal burned in the world. The number of cars on China's roads has increased fourfold since 2003, driving up demand for oil. Meanwhile, there is less and less space and respect for other species. For me, the most profound story of this period was the demise of the baiji – a Yangtze river dolphin that had been on earth for 20m years but was declared extinct in 2006 as a result of river traffic, pollution, reckless fishing and massive damming.

I switched my focus to environment reporting. It was not just the charismatic megafauna and the smog, though the concern about air quality never went away. It is really not funny to send your children off to school on days of high pollution with a cheery "Try not to breathe too much", knowing they will probably be kept in at break-times because the air outside is hazardous.

As I have noted at greater length elsewhere, I had come to fear that China may be where the 200-odd-year-old carbon-fuelled capital-driven model of economic development runs into an ecological wall. Britain, where it started, and China may be bookends on a period of global expansion that has never been seen before and may never be repeated again.

Developed nations have been outsourcing their environmental stress to other countries and future generations for more than two centuries. China is trying to do the same as it looks overseas for food, fuel and minerals to satisfy the rising demand of its cities and factories. This has been extremely good news for economies in Africa, Mongolia, Australia and South AmericaI sympathise with China. It is doing what imperial, dominant powers have done for more than two centuries, but it is harder for China because the planet is running short of land and time.

With their engineering backgrounds, President Hu Jintao (a trained hydro-engineer) and premier Wen Jiabao (one of China's leading experts on rare earth minerals) are probably better aware than most global leaders about the challenge this poses. While there has been almost no political reform during their terms of office, there have been several ambitious steps forward in terms of environmental policy: anti-desertification campaigns; tree planting; an environmental transparency law; adoption of carbon targets; eco-services compensation; eco accounting; caps on water; lower economic growth targets; the 12th Five-Year Plan; debate and increased monitoring of PM2.5 [fine particulate matter] and huge investments in eco-cities, "clean car" manufacturing, public transport, energy-saving devices and renewable technology. The far western deserts of China have been filled with wind farms and solar panels.

That is the most hopeful story of this grey era. If China could emerge from the smog with a low-carbon economy, it would be a boon for the world. But talk of the world's first green superpower remains as premature as the image of the "red menace" is outdated.

When my predecessor, John Gittings, left China after 25 years, he presciently foresaw how the the old cold war stereotypes would be shattered by the country's speed of development. But that is just the start of realignment. In the future, I believe the most important political division will not be between left and right, but between conservers and consumers. The old battle of "equality versus competition" in the allocation of the resource pie will become secondary to maintaining the pie itself.

But the transition has some way to go. In the next 10 years, China is likely to build more dams than the US managed in its entire history and, despite the Fukushima disaster, it plans to construct more about 20 new nuclear power stations. But even with this huge expansion of non-fossil-fuel-based energy, if the economy continues to grow at its current pace China will require about 50% more coal than it currently burns.I expect there will be a slowdown before then as overseas markets contract and domestic investment suffers from the law of diminishing returns. Long-hidden environmental costs – over-depletion of key resources and under-regulation of waste – will force their way onto corporate balance books and national budgets in the form of turbulent commodity prices and higher clean-up expenses. China may well look back on the Hu and Wen era as a golden age of growth and perhaps a missed opportunity to put in place the reforms needs to adjust to leaner times.

Respect, sympathy – and pessimism

Meanwhile a new leadership – almost certainly to be headed by Xi Jinping and Li Keqiang – will take the helm at this autumn's party congress. They will have their work cut out. While the Hu-Wen era was one of construction, Xi and Li will have to put more effort into maintenance. This will require more than the creation of wealth and construction jobs; it will require a system with greater flexibility, efficiency and a new set of values. I expect that transition will be more turbulent than anything seen in the past 10 years. But success or failure, I believe it will remain the most important story in the world.

So why am I leaving? Well, over the years, I have come to feel increasing respect, sympathy and affection for China, but also more pessimism. Journalists here are worn down like brake pads on a speeding juggernaut. Such cynicism is not healthy. I hope a change of scene will allow me to see China – and the world – afresh. Regardless of Beijing's choking smog, traffic and politics, it will be hard to match living and working in China. I shall miss hikes along wild stretches of the Great Wall, swimming under remote waterfalls, wandering through vibrant hutongs, incredible food and some of the smartest company I have encountered anywhere in the world.

In the past few weeks, I have said goodbyes to places, friends, long-suffering assistants and sources, some of whom have become global names as their country has risen in prominence over the past nine years: the environmentalist Ma Jun (winner of this year's Goldman Prize), the lawyer Teng Biao, journalist Li Datong, the rights activists Hu Jia (winner of the Sakharov human-rights prize), and Zeng Jinyan; the social network guru Isaac Mao, the public intellectual Wang Xiaoshan and the artist Ai Weiwei (recently named "the most powerful artist on earth").

On my final weekend in China, I went to Weiwei's grey-walled home in the Caochangdi art district. He was with his wife, two aides, a film crew and two lawyers, but as gregarious and mischievous as ever.

"It's hot. Let's take our clothes off," said Weiwei, who proceeded to strip to the waist. I was too shy to follow suit. So was everyone else. The lawyers simply proceeded with their brief about the next stage in his tax case.

I couldn't stay. The China story was moving on again. News had just come in that Chen Guangcheng was at Beijing airport, about to board a plane to the US. After six years of house arrest and prison, he was finally flying to freedom. I said my goodbyes and wandered home to write up what felt like an uplifting article to finish on. I knew though, that it was not really the end. For all the hardship Chen endured, I guessed he would miss China. I certainly will. This is a peak and perhaps one for mankind.

• Jonathan Watts will be based in Rio de Janeiro as Latin America correspondent from July.