The ecstatic poems of Jalal ad-Din Muhammad Rumi, a Persian poet and Sufi master born 807 years ago in 1207, have sold millions of copies in recent years, making him the most popular poet in the US. Globally, his fans are legion.

“He’s this compelling figure in all cultures,” says Brad Gooch, who is writing a biography of Rumi to follow his critically acclaimed books on Frank O’Hara and Flannery O’Connor. “The map of Rumi’s life covers 2,500 miles,” says Gooch, who has traveled from Rumi’s birthplace in Vakhsh, a small village in what is now Tajikistan, to Samarkand in Uzbekistan, to Iran and to Syria, where Rumi studied at Damascus and Aleppo in his twenties. His final stop was Konya, in Turkey, where Rumi spent the last 50 years of his life. Today Rumi’s tomb draws reverent followers and heads of state each year for a whirling dervish ceremony on 17 December, the anniversary of his death.

The transformative moment in Rumi’s life came in 1244, when he met a wandering mystic known as Shams of Tabriz. “Rumi was 37, a traditional Muslim preacher and scholar, as his father and grandfather had been,” says Gooch. “The two of them have this electric friendship for three years – lover and beloved [or] disciple and sheikh, it’s never clear.” Rumi became a mystic. After three years Shams disappeared – “possibly murdered by a jealous son of Rumi, possibly teaching Rumi an important lesson in separation.” Rumi coped by writing poetry. “Most of the poetry we have comes from age 37 to 67. He wrote 3,000 [love songs] to Shams, the prophet Muhammad and God. He wrote 2,000 rubayat, four-line quatrains. He wrote in couplets a six-volume spiritual epic, The Masnavi.”

During these years, Rumi incorporated poetry, music and dance into religious practice. “Rumi would whirl while he was meditating and while composing poetry, which he dictated,” said Gooch. “That was codified after his death into elegant meditative dance.” Or, as Rumi wrote, in Ghazal 2,351: “I used to recite prayers. Now I recite rhymes and poems and songs.” Centuries after his death, Rumi’s work is recited, chanted, set to music and used as inspiration for novels, poems, music, films, YouTube videos and tweets (Gooch tweets his translations @RumiSecrets). Why does Rumi’s work endure?

The inward eye

“He’s a poet of joy and of love,” says Gooch. “His work comes out of dealing with the separation from Shams and from love and the source of creation, and out of facing death. Rumi’s message cuts through and communicates. I saw a bumper sticker once, with a line from Rumi: “Out beyond ideas of wrongdoing and rightdoing there is a field. I'll meet you there.”

“Rumi is a very mysterious and provocative poet and figure for our time, as we grapple with understanding the Sufi tradition [and] understanding the nature of ecstasy and devotion and the power of poetry,” says the poet Anne Waldman, co-founder with Allen Ginsberg of the Jack Kerouac School of Disembodied Poetics at Naropa University, where she is a professor of poetics. “And the homoerotic tradition as well, consummated or not. He is in a long tradition of ecstatic seers from Sappho to Walt Whitman.”

“Across time, place and culture, Rumi's poems articulate what it feels like to be alive,” says Lee Briccetti, executive director of Poets House, co-sponsor of a national library series in the US that features Rumi. (It’s currently in Detroit and Queens and heads to San Francisco, Houston, Atlanta and Columbus in 2015.) “And they help us understand our own search for love and the ecstatic in the coil of daily life.” She compares Rumi’s work to Shakespeare’s for its “resonance and beauty”.

Coleman Barks, the translator whose work sparked an American Rumi renaissance and made Rumi the best-selling poet in the US, ticks off the reasons Rumi endures: “His startling imaginative freshness. The deep longing that we feel coming through. His sense of humour. There's always a playfulness [mixed] in with the wisdom.”

In 1976 the poet Robert Bly handed Barks a copy of Cambridge don AJ Arberry’s translation of Rumi and said, “These poems need to be released from their cages.” Barks transformed them from stiff academic language into American-style free verse. Since then, Barks’ translations have yielded 22 volumes in 33 years, including The Essential Rumi, A Year with Rumi, Rumi: The Big Red Book and Rumi’s father’s spiritual diary, The Drowned Book, all published by HarperOne. They have sold more than 2m copies worldwide and have been translated into 23 languages.

A new volume is due in autumn. Rumi: Soul-fury and Kindness, the Friendship of Rumi and Shams Tabriz features Barks’ new translations of Rumi’s short poems (rubai), and some work on the Notebooks of Shams Tabriz, sometimes called The Sayings of Shams Tabriz. “Like the Sayings of Jesus (The Gospel of Thomas), they have been hidden away for centuries,” Barks notes, “not in a red urn buried in Egypt, but in the dervish communities and libraries of Turkey and Iran. Over recent years scholars have begun to organise them and translate them into English.”

800 years ahead of the times

“Just now,” Barks says, “I feel there is a strong global movement, an impulse that wants to dissolve the boundaries that religions have put up and end the sectarian violence. It is said that people of all religions came to Rumi's funeral in 1273. Because, they said, he deepens our faith wherever we are. This is a powerful element in his appeal now.”

“Rumi was an experimental innovator among the Persian poets and he was a Sufi master,” says Jawid Mojaddedi, a scholar of early and medieval Sufism at Rutgers University and an award-winning Rumi translator. “This combination of mystical richness and bold adaptations of poetic forms is the key to his popularity today.”

The first of Rumi’s four main innovations is his direct address to readers in the rare second person, says Mojaddedi. “I think contemporary readers respond well to this directness.”

Second is his urge to teach: “Readers of ‘inspirational’ literature are drawn to Rumi’s poetry.” Third, “his use of everyday imagery.” And fourth, “his optimism of the attainment of union within his lyrical love ghazals. The convention in that form is to stress its unattainability and the cruel rebuffs of the beloved. Rumi celebrates union.”

Mojaddedi has completed his translation of three of the six volumes of Rumi’s masterwork, The Masnavi. It is, he said, “the longest single-authored emphatically mystical poem ever written at 26,000 couplets, making it a significant work in its own right. It is also arguably the second most influential text in the Islamic world after the Qu'ran.” The original Persian text was so influential that in Ottoman times a network of institutions was devoted to its study.

As new translations come into print, and his work continues to resonate, Rumi’s influence will continue. His inspiring words remind us how poetry can be a sustaining part of everyday life.

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.