If dramatists and explorers share an addiction to conflict, jeopardy and transgression, then Shakespeare and the first American settlers had rather more in common than their mother tongue. Since the founding fathers, this has developed into an association with an extraordinary cultural dividend. Four hundred years after the death of our national poet, and the subsequent landing of the Mayflower, the playwright who is an icon of Englishness has also become a central feature of the American dream, in which the mirror of his great dramas gets held up to a society perpetually in search of itself.

When President Bill Clinton says “our engagement with Shakespeare has been long and sustained: generation after generation of Americans has fallen under his spell”, he is acknowledging this most surprising fact – that Shakespeare’s afterlife as the greatest playwright who ever lived is now as much an American as a British phenomenon, and integral to the crazy fabric of life in the US.

American actors can better represent the sound of Shakespeare than their British counterparts

Stephen Greenblatt, the bestselling author of one of the finest recent Shakespeare biographies, Will in the World, finds many reasons for America’s adoption of the playwright. “One of the wonderful things about Shakespeare,” he says, “is that his plays are not exclusively a national possession. He can become an emblem of that which you might possess if you actually could not trace your roots to the Mayflower.” Then again, theatrical performance also implies liberation, and Greenblatt sees Shakespeare’s plays as speaking to the American need for reinvention. “The theatre is all about imagining yourself as other than you are.”

The works of Shakespeare have been an integral part of the American dream since the first settlement of the Thirteen Colonies, in part through the accident of timing. The Pilgrim Fathers set sail just two years before actors from Shakespeare’s Globe clubbed together to commission a volume of his plays. So the First Folio and the first Americans became informally twinned in a fortuitous assertion of originality. This cultural kinship was subtly reinforced by shared features in their accents. Shakespeare’s Warwickshire English and the emerging speech of the American colonists were both distinguished by the “post-vocalic” R, the rolled R sound in words such as “yard”, “car”, “hard” and “horse”. To this day, American actors can better represent the sound of Shakespeare than their British counterparts.

Greenblatt makes a double linguistic connection between Shakespeare and the US, a polyglot country from the beginning. “First, there is the history of American public eloquence that culminates in Abraham Lincoln not only quoting from the plays in his first inaugural address, but also using the cadences of Shakespeare to connect with his audience. Second, Shakespeare shares with the English language itself – his language – a remarkable openness to linguistic innovation by immigrant cultures. In that sense, Shakespeare is the quintessentially American author.”

The Pilgrim Fathers leave Delft Haven on their voyage to America, July 1620. Photograph: Print Collector/Getty Images

Shakespeare spoke to the first Americans in other ways, too. So far as we know, he never ventured on to the Atlantic, but he was fascinated by sea voyages and sinkings. Next to stabbings and sword-fights, drowning is the second commonest means of death in his plays. Some of his greatest works, notably Twelfth Night and The Tempest, begin with shipwrecked sailors. Stranded Viola’s “What country, friend, is this?” would have been a question familiar to generations of American settlers.

About 100 years after the landing at Plymouth Rock, reports of Shakespeare productions start to creep into the record, with Romeo and Juliet the most popular among new world audiences. By the mid-18th century, Shakespeare’s work had become a bible for Americans, and a treasured, felicitous reminder of their linguistic and cultural heritage. The first recorded performance of a Shakespeare play in the new world (Romeo and Juliet in New York City) took place in 1730. Colonial America was still in an early phase of its reverence for Shakespeare. According to American Shakespeare scholar James Shapiro, author of two books on Shakespeare, 1599 and 1606, “there were probably no more than 500 performances of all Shakespeare’s plays before the war of independence”. English puritans had been fiercely anti-theatrical, so it’s no surprise that their American cousins should view actors and acting with deep suspicion. As late as 1774, the Continental Congress was urging the people of Philadelphia to shun the “extravagance and dissipation” of the theatre.

The revolution changed everything. For a start, it brought to prominence a generation of Shakespeare lovers. On 14 July 1787, for instance, in the middle of thrashing out the details of the new constitution, George Washington broke away from the legislative haggling to watch a production of The Tempest. During the revolutionary war of 1776, a popular production of Othello in New York City saw black and white people mingling together with unprecedented freedom. Not for the last time, Shakespeare became the means by which discordant elements in US society could find harmony.

Now Shakespeare’s plays became part of the debate about what it meant to be an American, for example in a production of Coriolanus staged before revolutionary troops in 1778. Elsewhere, the first line of Hamlet – “Who’s there?” – explicitly raises the identity question. Meanwhile, plays such as Othello and Richard III allowed new Americans to explore new and troubling questions that would otherwise have been hard to articulate or admit to. Prospero’s island in The Tempest is more Caribbean and Mediterranean than American, but Miranda’s “O brave new world” echoes the thrill of landfall in the hearts of many first transatlantic explorers.

Miranda’s 'O brave new world' echoes the thrill of landfall in the hearts of many first transatlantic explorers

Americans might no longer refer to England as their mother country, but Stratford remained a popular tourist destination. When Jefferson and Adams visited in 1786, the author of the Declaration of Independence “fell upon the ground and kissed it”, and paid a shilling to see Shakespeare’s grave, while the future second president sliced “a relic” from the chair said to have belonged to the playwright. “Let me search for the clue which led great Shakespeare into the labyrinth of human nature,” Adams wrote in his diary. “Let me examine how men think.”

“Bardolatry” trickled down into all parts of the republic. When De Tocqueville toured the States in the 1830s, he described the immense and widespread popularity of Shakespeare. “There is hardly a pioneer hut,” he wrote, “in which the odd volume of Shakespeare cannot be found.” The plays, he reported, were almost as popular as the Bible. Shakespeare had become such good box office by the middle of the 19th century that the great showman PT Barnum tried, unsuccessfully, to ship “the birthplace”, a restored half-timbered house on Henley Street, back to the States.

This new American need to connect Shakespeare to the US would eventually sponsor replicas of the Globe in Dallas, San Diego and Cleveland, Ohio. Much later, the same instincts would inspire Sam Wanamaker to campaign for a reconstruction of the Globe on London’s South Bank, close to the site of the original.

Wanamaker’s vision is emblematic of the enthusiasm displayed by 19th and 20th-century American audiences for Shakespeare. The new world exhibited a far more sustained and widespread popular interest in his plays than the old country. This familiarity began in the classroom. According to Shapiro, it was a textbook such as the McGuffey Reader (1839) that sponsored American bardolatry, selling an astounding 122m copies in its first 100 years of school use. Possibly, it was classroom discussions of Shakespeare that disseminated his timeless themes among the population as a whole.

Occasionally, the passions inspired by Shakespeare got out of hand. In 1849, there was a riot, provoked by competing productions of Macbeth, in Astor Place, New York City, now the home of the Public Theatre. Bitter rivalry between the British star William John Macready and Edwin Forrest, an up-and-coming American actor, boiled over in the playhouse, spilled out on to the streets and led to a violent showdown between well-to-do Macready devotees and some American working-class Forrest fans. The National Guard was called out. In the ensuing melee, 20 people were killed and as many as 100 injured.

It’s hard to imagine, says Shapiro, that “people would come to blows over Shakespeare, with blood and killing, but that’s what happened. It was partly symbolic of a clash between British and American acting traditions, but it also dramatised a deepening conflict between highbrow and lowbrow theatre which was not resolved until Joe Papp and the Public Theatre launched “Shakespeare in the Park”, [free productions in Central Park] in the 1960s.”

Hip-hop show Hamilton is ‘Shakespearean in a lot of ways’. Photograph: Joan Marcus/The Public Theater/AP

Shapiro connects this tradition to the inspiration for Hamilton, a hip-hop rendering of American history that became a smash hit on Broadway in the summer of 2015. “Hamilton, which was developed at the Public Theatre,” he says, “is Shakespearean in a lot of ways. For me, it’s the closest I’ve ever felt to what audiences must have experienced at the Globe. Hamilton would be unintelligible to audiences not already steeped in Shakespeare.”

The Astor Place riots were unprecedented. In general, Shapiro sees Shakespeare in a more benign context as a lightning conductor for American political and economic culture. “There are so many things we cannot say to each other about our economic disparities, our political differences, and above all – race. Shakespeare offers an unofficial commentary on what Americans don’t want to talk about but which is still seething just below the surface.”

Debra Ann Byrd, who runs the Harlem Shakespeare festival, working with African-American and other mixed-race actors, believes that the passion for Shakespeare derives from politics as much as the schoolroom. For her, the plays become a medium through which the complexity of American society and its many unresolved tensions can be debated. When we met in the traumatic aftermath of the Charleston shootings, Byrd told me that “There has been a problem in America with racism and race relations. Somehow, for us, Shakespeare has become the medium in which we can have a conversation about race that addresses the conflict.”

The summer of 2015 was a dark time for the US with gun violence and the nationwide political hysteria surrounding the search for a Republican presidential nominee. At this juncture, Donald Trump was seen as less a threat than a diversion. Still, ordinary voters were plainly troubled by the state of the union, with President Obama widely blamed for failing to fix the race question. One play in particular seemed to speak directly to that predicament: Othello.

There has been a problem in America with racism and race relations – in Shakespeare we can address the conflict Debra Ann Byrd

In the course of my travels in search of America’s William Shakespeare, I lost count of the times in which ordinary citizens – taxi drivers; baristas; museum guides – spontaneously referenced Othello in conversations about the country’s race question, sometimes even quoting from famous speeches – lines such as “The quality of mercy”.

Turning to Julius Caesar, Othello or Richard III in popular discussion has old roots here in the US. President Lincoln, who first read Shakespeare as a boy in his log cabin, would later use the plays to connect with the voters. Privately, in the White House, he would read aloud from Hamlet and King Lear, getting his staff to perform the supporting roles.

In a moment of tragic irony, Shakespeare also played a bizarre part in Lincoln’s assassination. John Wilkes Booth was the brother of Edwin Booth, a popular and celebrated actor. The assassin identified Brutus in Julius Caesar as a role model in his struggle against tyranny. A letter written before the shooting in Ford’s Theatre appealed to the play’s authority. “But alas!” wrote Booth, “Caesar must bleed for it.”

Julius Caesar became part of an inner core – Hamlet, Macbeth, Othello, King Lear, Richard III, Romeo & Juliet, The Merchant of Venice – to which American audiences, and writers, returned repeatedly. By the 1870s, so well known were the plays that a humorist such as Mark Twain could place a Shakespeare parody at the heart of Huckleberry Finn without explanation.

After Twain, many American writers would make an imaginative rendezvous with Shakespeare. The catalogue of contemporary American writers who owe an explicit debt to Shakespeare includes John Updike (Gertrude and Claudius), Toni Morrison (Desdemona); and Arthur Phillips (The Tragedy of Arthur). Most prominent of all is Jane Smiley’s Pulitzer prize-winning A Thousand Acres, a novel that modernises King Lear and relocates Shakespeare to the American midwest. Her story recapitulates two centuries of mid-American Shakespeare.

William Shakespeare statue in Central Park. Photograph: Tetra Images/Corbis

Smiley recalls how she came to the playwright’s work: “Every year [in school] we read a Shakespeare play, starting with Twelfth Night. What I liked at first was the treasury of names in the dramatis personae, and what I kept on liking was the action and liveliness on stage – everything was interconnected. In graduate school, I also took a course in Shakespeare’s problem plays. When I returned to King Lear later on, the sense of Shakespeare attending to the mysterious past was important in my inspiration.”

At the outset, Smiley had a benign view of the playwright, with a vision of a man who was “mentally healthy, passionate but also funny, smart, balanced and essentially good. A guy I could relate to.” A Thousand Acres tested this relationship, but she nevertheless concluded her work with the feeling that “this sense of connection through Shakespeare with the distant past had been the loveliest reward of writing it. I had received a gift,” she says, “an image of literary history, two mirrors facing each other in the present moment, reflecting infinitely backward into the past and infinitely forward into the future.”

In New York City, deep in Central Park, Shakespeare’s statue embodies the ceaseless inner dialogue of the American dream. In the autumn of 1864, the statue had been the inspiration of the Booth brothers, proposed just six months before the assassination. For a moment, after Lincoln’s death, it looked as though the project would falter. Edwin, the senior Booth, appalled by his brother’s crime, was deeply ashamed of his connection. Eventually, however, le tout New York turned out for the inauguration of the statue. “If I had been betting on Shakespeare,” jokes Shapiro, “I would have betted against him. It didn’t have to go this way. But he gives us really good stories and larger-than-life characters whom we love. And a play like Julius Caesar will always speak to our republican values.”

In the decades after Lincoln, Shakespeare productions were a constant feature of frontier life in the west. When Los Angeles became the company town for an emerging film industry, Shakespeare shape-shifted again, becoming part of Hollywood westerns. He had been performed by companies up and down California for decades. In John Ford’s My Darling Clementine, starring Henry Fonda, the intinerant Shakespearean actor Granville Thorndyke, played by Alan Mowbray, performs “To be or not to be” to a bar full of desperadoes before grandly declaring that Shakespeare is not “meant for taverns or tavern louts”, a line that sparks an inevitable gun fight.

For actor Alec Baldwin, this scene is an emblematic moment in Hollywood’s fixation with Shakespeare. Baldwin, who shares this devotion, describes how, when travelling abroad, he will sometimes check into hotels as Thorndyke. Recently, offering a new TV show to HBO, he reports that inserting “Shakespeare” into his pitch convinced sceptical television executives to green-light the pilot.

Shakespeare is deemed not fit ‘for taverns or tavern louts’ in John Ford’s My Darling Clementine (1946). Photograph: Alamy

Perhaps the apotheosis of the marriage between Shakespeare and the new world occurs with West Side Story. The most famous 20th-century musical adaptation of Shakespeare began in 1949 when a friend of the director Jerome Robbins was offered the part of Romeo in a stage production and began puzzling about bringing the role to life. Robbins’s reaction, however, was more Shakespearean. “I said to myself: ‘There’s a wonderful idea here,’” he later recalled. Having begun to work on a story of ill-fated lovers and gang violence, Robbins approached Leonard Bernstein and the librettist Arthur Laurents. Eventually the young Stephen Sondheim came on board to write the lyrics.

Sondheim, speaking to me at home in Manhattan last summer, recalls: “I had no particular interest in the subject matter, but I knew I wanted to work with Bernstein, Arthur and Jerry. We never checked Shakespeare’s text. I don’t think any of us picked up that script once Arthur had written the ‘book’.”

Today, Sondheim has become a passionate follower of Shakespeare productions around the world, and especially in London. “I just can’t believe Shakespeare,” he says. “I mean, I can’t believe that one man did all this. Shakespeare is – oh, come on – some of his lines always make me cry.” Sondheim is less emotional about the relationship between West Side Story and Romeo and Juliet, a play in which, as he puts it, “so much happens”. He describes it as “a melodrama with something extraordinary happening in every scene. On top of the flavour, there is the compactness – the plot is always exciting.”

Shapiro shares with Sondheim a sense of anticipation at the imminent excitements of 2016. “I’m willing to bet,” he says, “that this year will be celebrated with more attention and more festivity in the US than in Stratford or London.” Shapiro cites the Folger Library’s plan to send a priceless copy of the First Folio from its collection for public display across every state in the union. “Of course,” he continues, “there will be a British invasion, but 2016 is going to be a year in which we see an indigenous American Shakespeare celebrated here.” Shapiro concludes our conversation with the suggestion that in the valedictory year of the Obama presidency, “Shakespeare remains at the centre of the things we disagree about. The issue of race is more contested than ever. The issues of leadership, belonging and national identity are Shakespearean questions, and Shakespeare is the place where we fight about ‘Who is an American?’”

• Shakespeare and the American Dream is on BBC Radio 4 at 9am on 19 and 26 April. The programme also broadcasts on BBC World Service at 10pm (BST) on 21 and 28 April as part of The Compass: Shakespeare in the World.