At 5am on a Sunday morning, Mohamed Diagayeté was disturbed by an urgent banging on the door of his house in Timbuktu, on the southern edge of the Sahara desert. It was a friend from the army: a heavily armed group of rebels had arrived at the city boundary, he told him; he'd done everything he could and must leave the city immediately. The soldier ran off to ditch his uniform and returned a few minutes later in civvies, intent on taking refuge in Diagayeté's house. Shortly afterwards, the first gunshots rang out over the city. "We could hear them firing. Bok! Bok! Bok," Diagayeté, an archivist, remembers. Before noon, a convoy of rebel pick-ups swept into the undefended town.

So began the 10-month occupation of Timbuktu, first by Tuareg separatist rebels, then by their fellow-travellers Ansar Dine (Defenders of the Faith), a jihadist affiliate of al-Qaida. It was a time of devastation in northern Mali: first the rebels pillaged the town, then the jihadis imposed a brutal form of sharia law on the population. Women were beaten for walking in the company of men. Music, a vibrant part of Malian culture that has been exported all over the world, was banned. Suspected thieves had their hands or feet chopped off after summary trials.

The largely moderate Muslims of Timbuktu were terrified. "When [the rebels] entered the city, people said if you were an artist they would cut out your tongue, because they hate music and want to ban it," Bintu Dara, a singer, tells me in the Malian capital, Bamako. "One of my cousins was beaten in front of me, given 100 lashes from the jihadis," she says. "My drum player was caught and put in jail. One of my relatives' sons was the first guy to have his hand cut off." Dara fled soon after, along with an estimated two-thirds of Timbuktu's citizens.

Timbuktu is a Unesco-listed world heritage site and the spiritual capital of sub-Saharan Africa; agonisingly, many of the cultural artefacts that gave the city its identity were destroyed or damaged. The shrines of Sufi saints were hacked to pieces and some priceless medieval manuscripts were burnt or stolen from the state archive. After the jihadists fled in the face of advancing French and Malian troops in January last year, the mayor of Timbuktu, Hallé Ousmane Cissé, revealed that the city's precious archive had been torched. What Cissé didn't know, however, was that, while several thousand manuscripts had been destroyed or looted, hundreds of thousands more had been smuggled to safety by an unlikely band of bibliophiles.

Abdel Kader Haïdara is a tall, 50-year-old librarian who wears a moustache and a pillbox kufi prayer cap. Over sweet mint tea in his office at the end of a red-earth road in the south-west of Bamako, Haïdara tells me the story of how he masterminded the smuggling of the manuscripts to safety from under the noses of the jihadists. "Before the hour of their arrival, we didn't think the rebels would come to Timbuktu," he says. "People were a bit scared but they didn't feel there was any great danger. They didn't make any kind of preparations.

"The first week of the occupation, there was a lot of shooting. The fighting was intense and everyone stayed in their houses." When he felt it was safe, he took a walk around the town and was shocked by what he saw. "I saw something that made me very, very afraid," he says. "I saw total insecurity. There were people of all ages looting the buildings, taking tables, chairs, air-conditioning units, anything they could find. What they didn't take they were smashing up.

"It was a great catastrophe. I knew that, if people continued like that, one day they would enter our library and smash up everything."

French soldiers on patrol in Timbuktu. Photograph: Fred Dufour/AFP/Getty Images

Timbuktu today is a sleepy collection of mud-brick houses that sits low in the ever-tightening embrace of the Sahara. Old men lead donkeys along sand-choked streets where children play barefoot and goats sift through the rubbish strung out along the roadside, eating everything they can find. But from the early 14th to late 16th centuries Timbuktu was famous for its wealth. It grew rich by virtue of its location at the northernmost bend of the river Niger, between the gold mines to the south and the salt mines in the Sahara. It has been estimated that in the 14th century two-thirds of the world's gold came from west Africa, a large proportion of it passing through Timbuktu, where it was transferred from the river to trans-Saharan caravans, from canoe to camel.

But Timbuktu was to become most famous as a centre of scholarship. In 1325, the fabulously wealthy emperor of Mali, Musa I, travelled to Mecca carrying a tonne of gold as spending money. "[He] set off in great pomp with a large party, including 60,000 soldiers and 500 slaves, who ran in front of him as he rode," one of the Timbuktu chronicles relates of Musa's hajj. "Each of his slaves bore in his hand a wand fashioned from 500 mithqal [about 2kg] of gold." On his return to Mali, Musa ordered the building of a grand mosque at Timbuktu. Another great mosque was added in the Sankore quarter of the city a few years later, and the surrounding area became a centre for Islamic teaching. The Encyclopedia Britannica states that by 1450 Timbuktu had a population of 100,000, a quarter of whom were students. Even if these figures are wildly exaggerated, Timbuktu was a thriving centre of learning, and manuscripts were highly prized: the traveller Leo Africanus, who visited in 1510, found books sold for more money than any other merchandise in the city's market.

Books reached Timbuktu by caravan from Fez and Cairo, Tripoli and Córdoba, and what the scholars couldn't afford, they would copy. Other documents were written in Timbuktu. The vast libraries that resulted included every subject: astronomy and medicine, law, theology, grammar and proverbs. There were biographical dictionaries, diaries, letters between rulers and subjects; legal opinions on slavery, coinage, marriage and divorce; the lives of Muslims, Jews and Christians; there were histories and poetry.

In 1591, the army of the sultan of Marrakech conquered the city. The libraries were looted and the most accomplished scholars killed or carried back to Morocco; the manuscript collections were hidden in holes in the sand, lost or destroyed on their way to Morocco. Tens of thousands more, however, were secreted behind the mud-brick walls of Timbuktu's family homes, to be handed down through the generations.

Abdel Haïdara now controls the largest privately held library of documents in the city, one he traces to a 16th-century ancestor; he also runs an organisation, Savama-DCI, that represents other private manuscript collections. Successive generations of Haïdaras have added to the family archive over the centuries, as his father did – by his own hand or with purchases made on his travels.

As the rebels approached, Haïdara knew the libraries would be vulnerable to looters: they were relatively large, prestigious buildings. So he began contacting families and told them to work out how to move their manuscripts into their homes. He bought steel lockers and, in the quiet of the afternoons when the jihadis were resting, the librarians and their assistants took the boxes to the libraries and began carefully transferring the manuscripts. "We brought them back to the family homes little by little," he says.

Mohamed Alkadi S Maïga describes how he worked: "People went to sleep after 2 o'clock in the afternoon. So between 2 and 4 o'clock, I went to the institute and we took the manuscripts out and put them into the bags. At night we looked for a push-push – a hand cart – and brought the bags to the house of our colleague." Over the next fortnight they made so many visits to the colleague's house that he feared they would be caught. "We realised we might be discovered by the Islamists because there was a lot of going back and forth, and then I got afraid and said we had to stop this," he says. He contacted the director of the state-owned Ahmed Baba Institute, who gave him a new house to transfer the manuscripts to, and the work continued. Maïga later moved an estimated 30,000 manuscripts for the institute.

His main concern was that the militants might think he was stealing. "I worried that if the jihadists caught me then maybe they would cut my hands off," he says.

Abdel Kader Haïdara with ancient manuscripts from Timbuktu packed into metal trunks. Photograph: Brent Stirton/Getty Images

By the end of April, almost all the private manuscript collections had been moved. "I didn't know 100% if they were safe, but I knew they were better there than in the library," says Haïdara. "I thought that very soon something would happen in the library and people would attack." Knowing he had done what he could, he joined the refugees crowding the battered 4x4s and trucks on the tracks south.

In Bamako, he joined forces with two other holders of large manuscript collections: Ismael Diadié Haidara of the Timbuktu Andalusian Library and Abdoulkadri Idrissa Maïga, the new director of the Ahmed Baba Institute. They gathered at Maïga's house to decide what to do next.

Early in June, Maïga decided to scope out the route the smugglers would have to take. He would send two "agents" north to try to bring back some hard drives and computers that had been left for safe keeping in the house of a colleague. One of these men was the researcher Mohamed Diagayeté.

A slender man with spectacles and a doctorate, Diagayeté is an unlikely looking agent. He had successfully fled Timbuktu with his family in April, and was now being asked to return to the occupied city in June. "We were worried," he says. "Anything could happen to us. But we thought we had to do it because it was our job. We had a mission."

From Bamako, they travelled to the edge of government-held territory at Sévaré, then on to Douentza, before taking the desert track north to Timbuktu, against the tide of refugees. They moved through territory controlled by different factions of militants, ragtag groups of young men armed with AK47s and clad in loose-fitting robes with metres of cotton wound round their heads as protection against sand and sun. At Douentza, they were stopped by the jihadists of Mujao, the Movement for Oneness and Jihad in West Africa. At Korioumé, where their route crossed the Niger, it was the National Movement for the Liberation of Azawad (MNLA). At the entrance to Timbuktu itself, the checkpoint was manned by al-Qaida in the Islamic Maghreb (AQIM), while the city was held by Ansar Dine. Diagayeté noted that the MNLA checkpoints were the most thorough and aggressive: they forced everyone off the vehicles and sometimes even searched their pockets. The jihadists were preoccupied with ensuring men and women were not sitting together.

The occupied city was lifeless, like "a body without a soul", says Diagayeté. "When you went to bed, you wondered if you would see out the next day. If you survived that day, you wondered if you would live through the following one. People spent all day sitting by the side of the street, doing nothing but hoping for good news from Bamako."

After a week, he left in a 4x4 bush taxi with 1,000 digitised manuscripts on a hard drive hidden in his luggage. Shortly after leaving Timbuktu, they were stopped at an MNLA checkpoint by rebels who demanded to search the car. As Diagayeté stepped out, he took the hard drive with him, then climbed back into the vehicle with it when the search was complete. It had been a close call: the rebels would have stolen it had they found it. From there they took off across open desert to avoid more rebel checkpoints on the main track south, joining the tarmac road 250km later at Boré. There were further checkpoints and searches in government territory, but Diagayeté reached Bamako with the digitised files in mid-June, and reported what he had found.

Soldiers man one of the many ­roadblocks on the smugglers’ route to Bamako. Photograph: Jerome Delay/AP

Buoyed by his success, the director sent more missions north. The first consisted of three agents, including Mohamed Alkadi S Maïga, who successfully brought back a locker and two sacks of manuscripts. This time it was the Malian army rather than the jihadis who gave them problems. "When I arrived in Sévaré, the military opened my locker and they asked me what it was and I said: 'Manuscripts!'" recalls the young courier. "'They are manuscripts of the state, so leave them alone!' They didn't believe me, but when I showed them my papers, and they opened the locker and saw it was true, they said OK and let us go."

The pressure to launch a full evacuation of the documents was building. In May, AQIM and Ansar Dine had begun demolishing the tombs of Timbuktu's Sufi saints and damaging its 15th-century Sidi Yahya mosque, where they broke down a door local people believed had to stay shut till the end of the world. Sixteen mausoleums would be totally destroyed. Haïdara resists discussing politics, but reports of the systematic destruction of the city's heritage could only have increased the urgency of the evacuation.

"We still hadn't started because we didn't have any money or any means, and because things were becoming very, very difficult in Mali," says Haïdara. The advice he received from people who had worked on cultural heritage in Iraq and Afghanistan was clear: they had to move the documents out of the city quickly. "By then things were starting to become critical in Timbuktu because the jihadi groups were becoming familiar with the population," he says. "There were even guides who started showing them around, pointing out which houses belonged to whom. And all this made me very worried."

They could delay no longer. Haïdara divided his team into three commissions, one in Bamako, one at the Timbuktu libraries and a team of couriers who would accompany each shipment. They communicated with cheap cell phones, which were discarded frequently to stop their signals being tracked. The couriers were there not so much for security as to smooth the transit of the lockers through government-held territory. "There were many checkpoints, and each time you had to open the lockers to show them what was inside, and that it wasn't ammunition," says Haïdara. "You needed someone with them who could open the lockers voluntarily, otherwise the soldiers would do it by force and damage the contents."

The shipping of the lockers was done by Timbuktu traders, who placed them on public transport, either by car or by boat on the Niger. As Timbuktu is hard to access even in peacetime (there is no tarmac road) the manuscripts travelled across the desert in four-wheel-drive bush taxis to Douentza and Mopti, where they were transferred for the journey down the tarmac to Bamako. "We put two to three boxes in each 4x4, then there were two guys who accompanied them," says Haïdara.

The size of the operation was astounding: according to the librarians, they had almost 400,000 manuscripts to move, in thousands of lockers, each of which was the size of a small trunk. Many of the shipments ran into difficulties, particularly at the northern end, where it was the Timbuktu commission's job to steady nerves. "Each time there was a little problem, we stopped everything," says Haïdara. When things had calmed down, they would quietly restart the shipments. "We moved a lot by the Douentza route, but we also moved a lot by the river. On the river it was different because two people could accompany 15 lockers in one boat. One boat could even take 100 lockers. But we thought it would have been too risky because if one boat sank we could lose everything. So we just put a few in each boat. We tried to be prudent."

The operation continued for the rest of the year, amid outbreaks of fighting between different rebel factions. Then, in mid-January 2013, the rebels launched a push south and threatened Bamako, triggering the intervention of France. The conflict reached a new pitch of intensity as the French military, supported by helicopter gunships and fighter jets, tried to force the rebels back. For Haïdara, this only made the evacuation more urgent: "The fighting was really intense, so we had to evacuate them as quickly as possible," he says. The war had brought transport across the desert to a halt, so they had to turn to the boats. In one major shipment, 15 pinnaces carrying manuscripts left Timbuktu together.

Malian pinnaces are sleek, timber-built cargo boats of traditional design: their prows and sterns bend upwards, like an upturned slice of melon rind. As Haïdara acknowledges, the convoy was easily visible – "fifteen boats is a lot of boats" – and someone was likely to notice. One night as they were making their way upriver, a French helicopter, suspecting they were rebels moving weapons or ammunition, swooped down to hover low over them, fixing them in the glare of its spotlight. One can imagine how the crews felt, transfixed in their journey up the dark Niger by a whirring war machine in the middle of a combat zone. After several minutes, the French helicopter peeled away, but it had been a close call. "The pinnace crew were very afraid," says Haïdara. "They called me the following day and I said from now on they must stop sailing at night. At 6 o'clock – sunset – they would have to stop. And I told them not to travel together."

The incident that unnerved Haïdara most occurred on an overland route that ran west of the Niger from Timbuktu. At a village near the town of Niafunké, a group of men with guns held up the vehicles and couriers. He still does not know what their motive was: "Perhaps they were bandits. Perhaps they were thieves who wanted to steal the manuscripts – who knows? When they called me, I called someone I knew in Niafunké. He called an imam in the village where they had been arrested." The gunmen wanted to take the couriers away, says Haïdara, but the imam intervened. "He spoke to them, negotiated with them and then they were allowed to pass. It was lucky that there was a network signal so they could speak on the phone, otherwise they would have been finished.

"They became very afraid, as did we. If something bad had happened the whole operation would have been ruined. But we were lucky and in the end they were more frightened than hurt."

A damaged manuscript at the Ahmed Baba Institute Photograph: Benoit Tessier /Reuters

The smugglers had not lost a single document, but there remained several thousand manuscripts they could not shift. These had been moved from the old Ahmed Baba Institute library to a new building in Timbuktu, which had recently been finished with South African funding. Because of its size and status, Ansar Dine were using it as a barracks. "The building had 15,000 manuscripts in it," says the Ahmed Baba Institute director, Maïga, "but it also had the jihadists. Luckily, about 10,000 of the manuscripts were kept underground in rooms that were really well hidden. Only about 4,000 manuscripts, which had been taken upstairs for restoration, were left out."

On 28 January 2013, Timbuktu fell to French and Malian soldiers. The departing rebels set fires in several buildings, including the new Ahmed Baba library. By the time the fire was brought under control, the floor was scattered with the charred remains of manuscripts. Timbuktu's mayor, Hallé Ousmane Cissé, who was living in Bamako, told journalists the manuscripts had all been burnt. "This is terrible news," he told the Guardian at the time. "The manuscripts were a part not only of Mali's heritage but of the world's heritage. By destroying them, they threaten the world."

In fact, more than 95% of Timbuktu's manuscripts were by then secretly sitting in Bamako. Maïga estimates 4,203 manuscripts belonging to the Ahmed Baba Institute were lost when the jihadists departed, either burnt or stolen. In the context of 400,000 the number is small, but far from insignificant. No one knows exactly what those manuscripts contained. "I think people were very affected by the loss," says Diagayeté, "because every manuscript is different. Some manuscripts don't have copies, and if they are gone, they are gone for ever."

The great bulk of the collections is now in hidden locations around the capital. Beneath Haïdara's office there is a blue-walled room filled with long tables laden with manuscripts. Here, restorer Eva Brozowsky works on preserving them, with funding from the German foreign ministry and the Gerda Henkel Foundation. The documents, she says, represent "a history and a kind of identity of the country… To keep them secret is the priority." Contemplating the scale of the restoration job, Brozowsky estimates 40% of the manuscripts are damaged. "There has been some long-term damage from insects, the sun, the climate, and some from the transport and the storage. There is some mould and water damage because they were hidden in cellars of houses."

To stop them decaying further, Brozowsky's team is cleaning the manuscripts and repairing tears in the pages with translucent washi paper made in Japan. The restored manuscripts are then placed in specially built containers made of acid-free, age-resistant alpha cellulose.

Maïga would like to see the manuscripts returned to Timbuktu, but first they need to raise more money. "They are in good shape here in Bamako. We had a problem with humidity but NGOs have provided dehumidifiers, which we put in all the houses where the manuscripts are, and since then there have been no problems."

It is possible, paradoxically, that the threat to the collections may end up aiding their cause. There is now a palpable urgency about digitising and cataloguing the documents, and according to Brozowsky this is the first time academic research has been done on the city's private collections, in part because Timbuktu is a difficult and often dangerous place to reach. Dr Michael Hanssler of the Gerda Henkel Foundation says 97-98% of the manuscripts have never been looked at from a research perspective. What does he think they will reveal? "You have a lot of natural science and mathematics in there. You have a lot of texts on the Arab languages. You have rhetoric and medicine. But some of the most interesting manuscripts are personal diaries, where people have jotted down stuff about their lives, stretching back 700 or 800 years. "

As to the documents' significance, it is "enormous", he says. "The Timbuktu manuscripts are probably the most important written tradition of west African history and culture there is. It's such a broad treasure it is hard to capture in a couple of sentences. People have been talking about them for more than 20 years, but no one has really had access to them."

The rescued manuscripts are just a small part of the story: there are many more scattered across the region. Thousands are being gathered in libraries in other cities, such as Djenné; others will remain gathering dust in chests in family homes where they have been stored for centuries. Some, buried in the Sahara and forgotten long ago, will keep their secrets for ever.