Years of painstaking work by scientists to expose a manuscript hidden for nearly a thousand years have shed new light on the genius of Archimedes, antiquity's greatest mathematician.

Known as The Archimedes Palimpsest, the manuscript is a Byzantine prayer book from the 13th century which was assembled using pages from several earlier manuscripts – one of which contained several treatises by the Greek mathematician Archimedes that were copied in 10th-century Constantinople. These were first discovered in 1906 by the Danish Archimedes scholar Johan Ludwig Heiberg, but as the text had been scraped away to make room for the prayer book he was only able to partially read them, and the book then went missing until it was auctioned – in a much more damaged state – at Christie's in New York in 1998. Bought by an anonymous American collector for $2m (£1.25m), it was deposited at Baltimore's Walters Art Museum, where scientists, conservators, classicists and historians have been working on uncovering the secrets of oldest surviving copy of Archimedes' works.

Using multispectral imaging and an x-ray technique which picked up the iron in the ink that had been scraped away, they discovered that Archimedes, working in the third century BC, considered the concept of actual infinity, something thought to have only been developed in the 19th century, and anticipated calculus. As well as seven treatises by the ancient Greek mathematician, including the only surviving copy of his The Method of Mechanical Theorems and Stomachion, new speeches by the classical Athenian orator Hyperides and a lost commentary on Aristotle's Categories from the second or third century AD were also found beneath the text of the prayer book.

The manuscript is now being displayed in an exhibition at the Walters, and Cambridge University Press is publishing later this month the two-volume book The Archimedes Palimpsest Project, which lays out the findings with images of the manuscripts, transcriptions of the texts and new readings of Archimedes' work.

The book's editor Michael Sharp called the discovery, in the treatise The Method of Mechanical Theorems, that at one point Archimedes considers the concept of actual infinity "very important for the history of mathematics and science". It sees Archimedes claiming that two different sets of lines are equal in multitude, although it is clearly understood that they are infinite, an approach which is "remarkably similar" to 16th and 17th-century works leading to the invention of calculus, according to the Walters Museum.

"The passage which makes this clear is one that Heiberg, the Danish mathematician of the early 20th century, had been completely unable to read. Since the concept of actual infinity has been crucial to the entire subsequent study of mathematics and physics, this is a particularly important new insight," said Sharp.

The palimpsest also contains the only existing copy of Archimedes' treatise Stomachion, in which he tries to discover how many ways 14 fixed pieces can be recombined to make a perfect square. The answer is 17,152 combinations. "Stomachion means bellyache – in antiquity you didn't call them brainteasers, you called them bellyachers. It's very interesting: not only is it completely different to his other works [but] it has been shown that it is actually the first work to develop the science of combinatorics – the maths of combinations which lies behind the mathematics of probability," said Sharp. "Before we knew this it was thought that combinatorics arose in the 17th or 18th century."