They stand by the sea at Lindisfarne. You’ll find them in the glens of Skye, in Slovenian gorges, on the peaks of Chamonix, Barbadian beaches, American national parks and Australian creeks. Stacks of stones, miniature monuments made by enraptured visitors and photographed for posterity are tiny tributes to human industry, our desire to make a mark, and our deep love of special places. But what appears a completely benign form of play in the natural world has become a plague.

Is there ever a right place? I’d say yes. Below the high-tide line on a well-visited beach, it's harmless and ephemeral

On a trip to visit his father in Orkney, John Hourston, the founder of marine campaigners the Blue Planet Society, was dismayed to find wild beaches dotted with stacks of stones. His tweets have triggered an unexpectedly heated debate during which he’s been belittled as a killjoy, a pedant, and even a misanthropist. Who knew meditative stone-stackers could be so angry? People have posted ghastly images of intrusive memorials from the pyramids to Stonehenge. Raging against the rocks is trivial and futile, say many. If stone stacks infuriate you, knock them down. Plenty of people do. Besides, what next? Ban sandcastles? Let’s focus on plastic in the oceans, people!

Of course, stone-stacking is an ancient and sometimes religious impulse. A cairn in Scotland is a memorial to someone lost, but also a pile of stones that marks a way through wild country: invaluable when the cloud comes down. Cairns keep us safe.

More recently, stone-stacking has become an art form and a competitive sport. Artists say the absorbing process of handling and balancing cool stones is meditative and good for our mental health. Children love it.

Blue Planet Society (@Seasaver) Almost every beach we visited in Orkney was spoiled by people leaving their mark and for a photo for social media. It's a worrying trend. It ruins the scenery and the environment. #CountryFile pic.twitter.com/vcDKELB1US

But what Hourston’s critics don’t seem to grasp is the almost industrial scale of this new age of stone-stacking. Adventure tourism and social media have created a perfect storm of stones. Cruise ships decant hundreds of visitors on to once remote islands such as Orkney, the Faroes or Iceland, each passenger burning with a creative desire to memorialise their sightseeing on Instagram. “Where do you draw the line?” wonders Hourston. “Orkney, Shetland, Iceland, Svalbard, or the Antarctic peninsula? We should start drawing the line now.”

Our personal monuments turn empty landscapes into peopled places. When we reach a remote summit or deserted beach, we know people have stepped there before, but for a moment we can enjoy a place where humans do not predominate.

No longer. A forest of stacked stones destroys all sense of the wild. Stacks are an intrusion, enforcing our presence on others long after our departure. It’s an offence against the first and most important rule of wild adventuring: leave no trace.

Stone-stacking in the wrong place can potentially mislead us on a misty moor. It can also harm wildlife. Birds such as oystercatchers make their nests on stony shores. These superbly camouflaged scrapes are almost impossible to see, and stone-stackers can destroy a nest in the breeding season without even knowing it. On other sites the moving of stones exposes soil and exacerbates erosion, destroying the cool undersides of stones that are sanctuaries for millions of invertebrates.

Hourston was particularly struck by the multitude of stacked stones on Skaill beach, immediately below the neolithic village of Skara Brae. It’s not fanciful to fear that ancient ruins may be disturbed. Historic England last year warned that stone-stacking was putting at risk historic monuments such as neolithic Stowe’s Pound in Cornwall. We can be prosecuted for stone-stacking – destroying an ancient monument – in the wrong place.

It’s an offence against the first and most important rule of wild adventuring: leave no trace

Is there ever a right place? I’d say yes. A stack of stones below the high-tide line on a well-visited beach is as harmless and ephemeral as a sandcastle. Just as there are graffiti walls, we could designate certain beaches or moors as permitted places for stone-stacking. Stone-stacking contests are obviously fun too.

Those who find stone-stacking self-absorbed and narcissistic may have a point. But stone-stacking in the wrong place is mostly just thoughtless.

Some will rail against more rules, or more self-restraint; but we need both, particularly when there are more than 7 billion of us. Sheer quantities of people turn inconsequential behaviour into acts with consequences. If we want to enjoy what’s left of our wild world, we have to be more aware than ever of our impact upon this Earth.

• Patrick Barkham writes for the Guardian on natural history