What was Timothy Leary really up to? We may soon know more now that the New York public library is buying 335 boxes of his papers, videotapes, letters and photographs for $900,000. Once it has spent 18 months to two years sorting them out, the collection will be available to the public.

These papers are not just the rants of this decidedly peculiar man – the 1960s drugs guru whom Richard Nixon called "the most dangerous man in America". There is correspondence with the likes of Allen Ginsberg, Cary Grant, Aldous Huxley, William Burroughs, Jack Kerouac and Arthur Koestler.

Perhaps these papers will give a glimpse of great genius arising from the clash of creative minds with powerful drugs – of insights gained and mystical peaks reached. Or perhaps they will show the horrors and mental decline of drug abuse and excess.

Possibly the most interesting will be the numerous "session records", that is, descriptions of taking LSD, psilocybin, mescaline and other psychedelic drugs. These will presumably give a more realistic picture of what these poets, writers, professors and actors actually experienced at the time.

Leary's is a sad story. A professor at Harvard, he took his first mushroom trip in 1960 and declared that he learned more in the following five hours than he had done in 15 years of study and research in psychology. This experience led to the Harvard Psilocybin Project, which Leary ran along with Ralph Metzner and fellow professor Richard Alpert. It was Alpert who subsequently swapped drugs for Eastern religion and became Baba Ram Dass.

In 1962 Leary took LSD and reportedly had "the most shattering experience of his life". This new, purely synthetic drug seemed to reveal previously hidden realities and he wanted to share his discoveries with the world. Yet his own world began falling apart. Having claimed he had given LSD (which was then legal) to hundreds of Harvard students he was eventually sacked for not turning up to teach classes.

He was later convicted of possessing marijuana and sentenced to 10 years in prison. He escaped from a high security jail and fled with his wife to Algeria and then Switzerland but was finally arrested in Afghanistan and returned to prison for three more years. Once free, he devoted his undoubtedly extraordinary mind and the last 20 years of his life to virtual reality, programming and cyberculture. When dying from prostate cancer he worked with friends to document the whole messy process. He died in 1996 and a portion of his ashes was launched into space.

Leary believed that psychedelic drugs, used at the right dosage, in the right company and setting and with appropriate psychological support, could provide better therapy than any conventional method, and even provoke magically rapid transformations. He explored the use of psychedelics to treat alcoholism and other addictions, and worked in prisons to use them as a means of reforming prisoners' lives. Many of his research participants reported mystical or spiritual experiences, and claimed that their drug experiences permanently changed their lives for the better.

We now know that these claims are far from crazy, and that psychedelics have tremendous potential for good as well as harm. The tragedy is that Leary's own actions contributed to the disaster of drug prohibition. On 6 October 1966, LSD was made illegal in the US and was so tightly controlled that not only were supply and possession made crimes but all of the legitimate research programmes were closed down. Not only was this extraordinary drug demonised and access denied to everyone who might have benefited from it, but also even researchers were prevented from learning anything more about it.

Arguably Leary himself was responsible for wrecking any chance that psilocybin or LSD could become respected and well-used drugs. Possibly if he hadn't got so carried away, so drunk with celebrity, and so successful at spreading the catchy meme "Turn on, tune in, drop out" we might now be living in a better world. Nothing can now wipe away those disastrous decades of prohibition, even though they may now be nearing their end, but perhaps these papers will help us better understand how it all came about.