George Clooney thinks Britain should return the Parthenon marbles to Greece. It's a widely held and perfectly respectable view – certainly not a "Hitlerian agenda" for London's cultural treasures, as Boris Johnson would have it. But is it right?

There are certainly bad reasons to return the marbles.

One is that Lord Elgin, the British ambassador to the Ottoman empire in the early 19th century, denuded the Parthenon of much of its sculpture immorally, or even illicitly. He certainly seems to have exploited his firman or licence from the Sultan to remove "stones with inscriptions and figures" from the building with an enthusiasm that did not escape the critical notice of contemporary observers.

When Elgin was forced by straitened financial circumstances to sell his booty, and the British parliament voted in 1816 to accept a select committee report that he had legitimately acquired it before purchasing the marbles for the British Museum, it took a narrowly legal view of the issue that was not universally accepted at the time, or even in the house: during the debate, Hugo Hammersley MP suggested that they be returned to Athens.

But we can't determine the right thing to do now solely on the basis that someone did the wrong thing done in the past – whether Lord Elgin himself, the government who endorsed his actions, or the British Museum staff who damaged the sculptures through over-enthusiastic cleaning in the 1930s. That's not to say that institutions and states, and perhaps even families, shouldn't be held responsible for the present consequences of their past actions, even if none of the people directly involved or affected are still alive. But putting those past wrongs right is rarely as simple, or as cheap, as reversing the original act.

Another reason is that the marbles belong to Greece. The temple itself was built in the 5th century BC by the city-state of Athens for Athena, its patron goddess, and it housed the tribute the Athenians received from the other city-states subject to them: hardly a symbol of Greek democracy or fellow-feeling. Athens ceased to exist as a Greek polis in the 6th century AD, well over a thousand years before the seventh Earl of Elgin removed them from the Ottoman empire. The nation of Greece dates back to 1830, 20 years after Elgin finished his dirty work, and 14 years after the British state turned them over to the British Museum, whose trustees are now the legal owners of the sculpture. It is hard to see what the modern nation states of Greece and the UK, or the issue of ownership, really have to do with the question of where the Parthenon sculptures are best displayed.

But there's a very bad reason not to return to the marbles, which is that doing so would set a precedent. That depends entirely on what is done. Returning the marbles to Greece would indeed suggest that cultural artefacts should as a rule be sent to the modern nation state occupying the land on which they were built or found.

But if the guiding principle is that our global cultural heritage belongs to all of us, and should be available to as many of us as possible, then more difficult decisions have to be made.

In this case, there's a persuasive argument that people should have the chance to see the marbles beside the Acropolis on which they were first erected. In the new Acropolis Museum, the Parthenon itself is visible through the windows of the room in which the marbles would be displayed together with the fragments that remained in Athens. The sculptures currently split in two – including a decapitated goddess and a great procession that disappears half way through – would be reunited, and would finally make all their sense. Athens is no less accessible than London to the rest of the world, and to see and think about this temple and almost all of its sculpture on the same morning, under the same Athenian sky, would be a privilege and a joy.

So perhaps George Clooney is wrong, and Britain should not return the Parthenon sculptures to Greece. But there is a case that the British Museum should send them to the Acropolis Museum in Athens with ownership invested in a trust connected to neither institution or state. The fact that this solution would be unacceptable to both Greek and British governments is a problem of modern nationalism, not ancient sculpture.