Several times in the past 10 years scientists have had to rewrite the textbooks on Neanderthals, the last species of human to go extinct. Once the archetype for primitive, uncivilised behaviour, the species, illuminated through fossil excavations and lately analysis of their genome, has emerged as being not too dissimilar from our own.

Contrary to their dim-witted image Neanderthals have been found to have used tools, to have worn jewellery, and, lastly, to have interbred with our Homo sapiens ancestors to such an extent that 4% of every modern European's genome is traceable to Neanderthal origins.

Now comes what could be the final nail in the coffin of the "unintelligent Neanderthals" myth: they might have been the first human species to paint in caves.

Using state-of-the-art techniques scientists have dated cave paintings at 11 locations in north Spain, including the Unesco World Heritage sites of Altamira, El Castillo and Tito Bustillo. Samples from 50 paintings of different styles were collected and the scientists discovered that a red disc on the wall of the El Castillo cave had to be more than 40,800 years old.

"This is currently Europe's oldest-dated art, by at least 4,000 years," said Alistair Pike, of the University of Bristol, who led the research. "We know the modern humans arrived in Europe between 42,000 and 41,000 years ago."

Nearby hand stencils, formed by blowing paint against a hand pressed against a cave wall, were at least 37,300 years old. The results are published on Thursday in the journal Science.

Neanderthals arrived in Europe about 250,000 years ago and, if the dates of the cave paintings at El Castillo are correct, it is probable that they made them. "Perhaps we should start thinking of [Neanderthals] as the European brand of Homo sapiens, morphologically different from what we call the modern humans in Africa," said João Zilhão, of the University of Barcelona, an author on the Science paper. "But they were sapient people as well – that is probably the implication of the last decade of results."

Already, symbolic culture has been shown to have existed among the Neanderthals, with 50,000-year-old evidence indicating body painting and jewellery decoration made from bones, teeth, ivory and marine shells, at sites in France.

"It would not be surprising if they were indeed Europe's first cave artists," Zilhão said. "In the context of what we've learned about Neanderthals in the last decade it really should not be very surprising."

If Neanderthals were decorating their caves, it might mean that language and advanced cognition were present in the human lineage further back than suspected – perhaps since the time of Neanderthals' and modern humans' last common ancestor, who lived at least half a million years ago.

Building a reliable chronology for cave art is difficult because the primary dating method, the radiocarbon technique, is not suitable for engravings or paintings made purely with mineral pigments. And, the availability of just tiny samples means the effects of contamination are highly magnified.

"So some radiocarbon dates disagree with other radiocarbon dates on the same painting or even dates that are processed in different ways that give you a different radiocarbon date," said Pike.

Zilhão and Pike used a different method. Over time the paintings had built up a thin crust of calcium carbonate, formed by the same process that gives caves stalagmites. The crusts held radioactive uranium, the element that decays to thorium. Measurements of the buildup of thorium could reveal how long ago those crusts formed and, since they were on top of the paintings, indicate a minimum age for the drawings.

Uranium-thorium dating has been in use since the 1960s but has usually required hundreds of grams of calcium carbonate. The team working with Pike and Zilhão was able to refine the method, using only a few tens of milligrams, a sample about the size of a grain of rice.

To prove definitively the involvement of Neanderthals in cave art creation, scientists will need to find and date samples from more caves using the new technique. If there are paintings pre-dating the arrival of modern humans into Europe – that is, older than 42,000 years old – it will be hard to doubt that Neanderthals were the world's first cave artists.

Pike said that the most exciting thing about the possibility that the El Castillo cave had Neanderthal art was that anyone could walk in and see a Neanderthal hand on the wall. "And this is something that had been invisible to archaeology until we worked out where to look."

• This article was amended on 19 June 2012 to correct a description of Neanderthals as "the latest species of human to go extinct". Neanderthals are the last species of human to go extinct.