In 1607, a galleon called the Susan Constant arrived in the New World from Blackwall, London, and established the first permanent English settlement of Jamestown, Virginia. Over the river in Surry County lies the land John Rolfe received as a dowry when he married Pocahontas. Today, up the road in Yorktown, the children of the fife and drum society make their evening parade around a monument marking the last great battle of the American Revolution, in front of a family-run bed and breakfast called the Hornsby House Inn. In Colonial Williamsburg, the heart of this historic triangle, distant cannon fire can be heard and every so often someone runs past in a tricorn hat.

A few years back, Williamsburg’s best-known modern-day son put out an alternative settlement narrative for Virginia in the form of a punk rock sea shanty called “The Black Rats of London”, in which he pointed out that it wasn’t divine intervention that toppled the natives but “parameciums” from “imported English dirt”. Bruce Hornsby is the first person I’ve heard use the phrase “American Holocaust” in casual conversation but then his family goes back a long way. At the inn, his cousins explain that their five-times great-grandfather, Nathaniel Bowditch, wrote The New American Practical Navigator, the oceanography bible still carried on every US naval vessel.

Three hundred and seventy-nine years after the rats, London sent something else to the New World, in a roundabout way: a five-minute song with two jazz piano solos that was considered too out there by American A&R men to release as a single but caught the attention of a BBC Radio 1 producer, Mik Wilkojc, and became an international hit. Everyone knows “The Way It Is” (“That’s just the way it is/Some things will never change”), its air-punching riff off-set by an apparently gloomy prognosis for American race relations – though younger people think it’s by the rapper Tupac Shakur, who sampled the riff and the chorus in “Changes”. Bruce Hornsby and the Range hit the MTV video boom in the same year as Peter Gabriel’s “Sledgehammer” and Madonna’s “Papa Don’t Preach”. He was a straight 31-year-old in a white shirt with a strange ringlet ponytail and a Donald Fagen-ish voice of the kind Amazon now calls “adult contemporary”.

In fact, Hornsby may be the only musician apart from Frank Zappa to have had a pop moment, a rock moment, a country moment (he won the bluegrass Grammy in 1990 and “pissed off all the purists”), a jazz moment (ongoing) and a modern classical one: the “12-tone pop” he’s writing now is a strange mix of Schoenberg and Broadway. He is also one of just two keyboardists from the Grateful Dead still vertical and above ground.

I first saw him do a solo piano show in Troy, New York, the day after Hallowe’en last year. The city was hit hard by the recession in the 1990s and still hasn’t recovered. Many of the shops in town were boarded up and there was no one on the street. I sheltered from the incoming snow in the CVS pharmacy, which was where, as it turned out, the rest of the town seemed to be hanging out too. As the sky darkened, I couldn’t help but wonder why a guy turning 60 and still bringing in royalties from half a dozen hits he’d bagged in pop’s most lucrative era was putting himself through this – which Ramada Inn he was sitting in, waiting for night to fall. Hornsby gives his albums away with his concert tickets now. As he pulled up to the venue in a small, black car with no staff and no props, I thought of his fellow 1980s piano man Billy Joel being helicoptered from Long Island to Madison Square Garden to play his old tunes for $2m a night.

“Y’all may be used to this temperature but I am a pathetic southern weenie and I am not,” Hornsby said, seating himself at a Steinway in jeans and sneakers. He is 6ft 4in and moves slowly, a bit like an athlete conserving energy. “I’m serious. I was left standing at stage door for, like, 23 seconds.”

In 2009, the director Bobcat Goldthwaite cast Hornsby in the Robin Williams movie World’s Greatest Dad, a black comedy whose theme – auto-erotic asphyxiation – was only marginally less taboo than that of his previous film, about a woman who had sex with a dog. Hornsby was a running joke in the script, his music loathed by the teenage central character (a furious masturbator) and loved only by his sad-sack dad. He once described his audience as a collision between two groups: Dreadlock Dave and Stockbroker Stan, the first being the tie-dyed, bearded, jam-band crowd he picked up during his time with the Grateful Dead, whom he joined aged 35 in 1990, and the second a nostalgic audience who bought his first two records in an era of yuppie pop crystallised in American Psycho, when Patrick Bateman murders a fellow banker to the sound of “Hip to Be Square” by Huey Lewis and the News. (Huey had a number one hit with Hornsby’s song “Jacob’s Ladder”.)

Here’s how it works. The gigs are request shows – write a song title on a bit of paper and take it up to the stage. At some, there are so many requests that it looks like someone has kicked over a trash can. The size of his hard-core fan base varies from state to state; he notes that he’s not very big in the deep south (“about five foot two”). Request your hit, many of them branded on the American psyche – the rock lament “Mandolin Rain”; or “End of the Innocence”, the critique of the Reagan administration that he co-wrote with Don Henley – then surrender your desire to hear it as you want to hear it. Expect ten-minute detours: along with Bud Powell’s “Tempus Fugit” and extracts from Olivier Messiaen, he has made diversions into “Three Blind Mice” and “There’s a Hole in My Bucket”. Expect sarcasm if you request something he doesn’t play any more. And don’t leave your mobile phone on. He has been known to get his own back on an inattentive audience by playing someone’s favourite song in two different keys at the same time.

The solo tour – the most gruelling part of his repertoire by his own admission – is roughly a fifth of the Hornsby workload. It usually takes place between April and November (November to March is the basketball season, when he stays put). Then there’s the ongoing bluegrass collaboration with Ricky Skaggs; his musical SCKBSTD (“sick bastard”); his rock band the Noisemakers, now in their 17th year; the 20-year gig writing music for Spike Lee films; and the folk music (his next album is “your long-awaited dulcimer record”). He will join the Grateful Dead for their last ever gigs in California and Chicago this summer, playing to 380,000 people. “You know about that?” he says. “Mental.”

We’re at his house in Williamsburg, a white weatherboard number his brother built for him when he moved back here four years after the big hit. It is set on a tidal creek and looks a bit like an Edward Hopper painting, which is appropriate since he has some: Farm House at Essex (currently on loan somewhere), Sultry Day and Funnel of Trawler, and many sketches, including train and waiting room scenes such as the enigmatic Chair Car, covered in Hopper’s annotations. Hornsby’s grandfather, an organ player, knew Hopper as a young man. His house lies in a patch of land he has named Tossers Wood, complete with a sign that his wife made him – his homage to the phrases of the Old World. He was pleased on a recent visit to London to learn the rhyming slang “thrupennies”.

He presents me with his left index finger to shake; too many firm grips over the years could damage his hands. I ask him why he drives round to his gigs on his own and he says, with an amused look, that it suits him as he gets further into his misanthropy. Then he walks me through the house showing me pictures of his identical twin sons, Keith (after Jarrett) and Russell (after Leon), and digs out an ancient Casio synthesiser, fiddling with its batteries and bashing out the riff from “Shadow Hand”, his song about a man with imaginary friends. He is a big, rather stately figure in trackpants, given to bouts of Tigger-ish excitement. It is difficult to see this Hornsby in the one from 1986, furrowed of brow and tightly rolled of sleeve, looking less than comfortable at the peak of his sudden fame.

“When we first came out in the UK, oh, it was amazing,” he says, settling into a tiny chair and somehow managing to sit cross-legged. “We broke worldwide and it was a very intense year. I had to learn how to be a public presence right away. The head of publicity at RCA said, ‘Bruce, here’s some press,’ and handed me a pile of Xeroxes from Melody Maker and NME. One review said: ‘[English accent] This particular gherkin calls himself Bruce Hornsby. With any luck we will never hear from him again!’”

He describes the moment he got famous as “the least enjoyable year of my career”. The ascent had been slow. He can be seen in the video for the world’s rudest song, “Sugar Walls” (“Come inside my sugar walls”), written by Prince and performed by his protégée Sheena Easton. Hornsby toured with Easton for two years playing keyboard in the mid-1980s. He says that his son Keith, a college basketball star, watches the video whenever he has a bad game.

“The Way It Is” was the second song he released and it remains his biggest hit. At the time, he’d been living in LA with his wife, Kathy, and his brother John (who co-wrote many early tracks) for six years, trying to make it. They had moved from Williamsburg with $13,000 in savings; she worked as a junior school teacher and he got a salaried position as a songwriter for 20th Century Fox. He had (very sensibly) given himself ten years to get a record contract; if that didn’t work, they were coming home. When he finally got one, he turned down the offer of a $40,000 advance in return for half his publishing rights and “took a gamble on myself”. The small family industry took place against the backdrop of the Sunset Strip and hair metal. Hornsby worked the classic rock clubs – the Roxy, Madame Wongs, the Lingerie. “We were doing just fine,” he says. “In the middle of our year, we were a theatre act playing 2,500-seaters but then, by the summer of the next, it was 10,000.”

“It was just too much. Such a full onslaught of attention and opinion. I never thought it was going to be that big a deal. I thought I was terrible at all of it and I was. I see old clips and frankly, that thing about youth being wasted on the young? Only I wasn’t even a youth. I was 31 but my level of ability at all this extra-musical business, interviews and how to act – I was 21. I felt like an idiot so much of the time. I thought, is this what this is about? If so, I want off.”

I ask him to explain why “The Way It Is” was so successful. He says he can’t possibly do that, then does. “I see it as a novelty record,” he says “There are things that set it apart. I feel the same way about ‘Sultans of Swing’ by Dire Straits. It goes down easy and isn’t that what a lot of pop is about? But at the same time, it’s a completely different sound than you’d heard. Even the big piano guys like Elton and Billy Joel, they didn’t really solo like that. A pleasing sound with solos. Like Mark Knopfler on ‘Sultans of Swing’. That’s how I explain it. But that’s complete crap, too, probably.”

Like other “novelty” mega-hits – Journey’s “Don’t Stop Believin’”, with a chorus that doesn’t come until most of the record is over, or “Bohemian Rhapsody” with its opera section – “The Way It Is” demanded more of the ear than regular songs; proof, perhaps, that the guy stuck in traffic with the radio on picks up, as the songwriter Eg White once told me, on the deliberate dropped stitch, missed beat or melodic twist that marks a record out from the mainstream. Hornsby has always said he couldn’t believe some of the things he managed to slip under the wire on the top 40 but the hits didn’t last. He moved back to Williamsburg in 1990 and two years touring with the Dead threw him back into improvisation (they could play one song for an hour). As his chart presence faded, his critical standing increased. For subsequent records, he pulled in everyone he wanted – Pat Metheny, Béla Fleck, Ornette Coleman, Charlie Haden, Chaka Khan, Eric Clapton. Bonnie Raitt tells me she considers Hornsby to be set apart from other artists by his “level of devotion to his instrument, to his writing and to the sheer joy of being totally spontaneous onstage”. Two years ago, Elton John said that Hornsby’s piano playing on Raitt’s 1991 hit “I Can’t Make You Love Me” made him “seek perfection. It is sublime. He is one of the best pianists – if not the best – out there.”

****

The College of William and Mary is the second-oldest university in the US. In its library can be found bound copies of a periodical called Piano Monthly, which Hornsby edited between the ages of 17 and 20 with his brother and two friends. One afternoon, he slipped several copies into the education section between Phys Ed Weekly and the Princeton Review. They later migrated into the music periodicals along with the jazz title Downbeat and finally, after he’d had his hits, they were placed in “rare books” alongside the Thomas Jefferson papers.

Piano Monthly was not about pianos. The first cover featured two all-star wrestlers; another, a Vietnam commando (the “Blood & Guts” issue). There is an advert that reads: “Learn the harpsichord in seven days and impress the ladies.” There are reams of adolescent rock dreams – phoney reviews of records that appeared in the hazy early 1970s [“It’s been quite a while since Harry Nilsson spent hours in a cubby hole with Bruce Hornsby licking and pasting labels on his albums”]. In a telling moment, Hornsby reviews a release by the New Orleans piano titan Professor Longhair and irony is suspended. “Longhair can probably do more with the 1-4-5 progression than any other piano player.”

At this point, he had been playing piano for under a year. Within two, he was at the prestigious Berklee College of Music in Boston. I ask him how this happened and he says Berklee took anybody in those days. His father, who ran a real estate business, told him he must pay his own way through college if he wanted to do music, so he played cocktail jazz in hotels at night. Aged 20, he failed to get into the New England Conservatory because he chose an impossible audition programme of Chick Corea and McCoy Tyner, which he “had no business playing”. Then he bought an upright piano, moved out to a farmhouse in Toano, Virginia, and practised for six months, eight hours a day. He entered the University of Miami School of Music to study jazz in 1975 at the end of its “golden era”; in the years above or already teaching were Jaco Pastorius, Pat Metheny, most of Metheny’s first band and the guitar whizz (and future member of Deep Purple) Steve Morse. These were the fragrant, long-haired legends of 1970s jazz rock. Something tells me Hornsby didn’t cut the same figure.

“I was a geek! Oh, totally. I looked hilarious,” he cries. “But it’s a beautiful reason why I was a geek and why anyone is a geek. I was so consumed with it. You are so interested in what you are doing that you don’t even think about the fact that you’ve worn the same white T-shirt for a month without washing it and the same jeans and you probably reek from 20 yards away.”

But this was 1970s Miami, I say, the hippest place on earth. “You can have your own little world,” he says. “You can be completely oblivious to, I guess, what was happening then – the TK Records phenomenon, the disco era, KC and the Sunshine Band, George McCrae, [sings] ‘Rock me baby!’ I stayed in the practice room. And at night I earned money playing in a band with a gay Spanish singer who shook it while we played: [sings] ‘That’s the way, ah, ah, I like it!’ By day I was in classes, then the practice rooms. So really, I had no time to know what was happening in Miami.”

Hornsby’s ascetic streak didn’t die with his student days. “It was around Christmas time. I had just turned 40,” he tells me. “I thought, right, what am I going to do now? Most of my singer-songwriter friends kind of stop at that point. They’re not interested in improving on their instruments and that’s fine, I get it. So many of the people who were lionised in that era of pop music I came out of, chances are they don’t even play any more. I thought, what am I going to do?”

“I used to find myself in solo piano contexts occasionally and I always felt inadequate. There’s this area of piano playing that has always been represented for me in its most amazing, fullest form by Keith Jarrett, which is the independence of the hands – being able to ‘split your brain’ as an improviser. There were things Jarrett could do and I’d think, this is so amazing, I can’t even sniff at it, I can’t even start to replicate this. So I decided to rededicate myself to the study of the piano aged 40, around holiday time, when the boys were out of school. Kathy was pretty bummed with me. She said, ‘You spend enough time out there as it is.’”

I ask him to demonstrate what he was teaching himself to do – the piano equivalent, he says, of “rubbing your stomach and patting your head”. We go next door to his studio and he starts thundering away on a complicated left-hand passage from a 1998 song called “King of the Hill”. With his right, he hits one note repeatedly on the beat, then turns it into triplets, then syncopates it. “You have to crawl . . . before you can walk . . . before you can run,” he shouts. I feel like we’re in a ballet class. By the end, he is playing a completely different rhythm and melody with each hand. I ask him how long the passage took him to learn and he says a couple of months.

“People say, ‘Ah, you just want to show off,’” he says, rising from the piano stool and floating back into the other room. “That is always the argument about virtuosity. The punk aesthetic dictates that it shouldn’t have to be about playing your instrument well; it’s all about the emotion and the spirit of it. And I get that. I don’t dismiss it. The accordion is my punk instrument, because I’m terrible at it but it doesn’t stop me. I don’t mind sucking on the accordion in front of thousands – same with the dulcimer.

“But to dismiss virtuosity as musical masturbation is to dismiss Evgeny Kissin, the great classical pianist, playing Rachmaninoff as a show-off. I know Joni Mitchell used to really take umbrage with this mindset because she had Wayne Shorter playing with her and people would dismiss him as this ‘jazzbo’. Shorter was not a guy about musical masturbation. He is a guy who will play one tone for 20 seconds. If you want to limit the range of the colours you can paint with, then be my guest. But to me, I just think it’s too bad.”

On the piano, laid out in preparation for the second leg of the solo tour after the basketball season, is a copy of Caténaires by the New York atonalist Elliott Carter (1908-2012). It is a mad, mad piece – a terrifying scud of black semi-quavers laminated and stretched out over umpteen pages. Hornsby finds Carter inspiring because he composed until he was 103. What does Stockbroker Stan think when he requests “Every Little Kiss” and gets Elliot Carter instead?

“Well, look, I’m sure he hates it!” he says. “And I totally understand. I feel his pain. But I just can’t let it stop me from the pursuit of the new and the pursuit of developing myself further. No scientific study has been made of this but I am almost sure that what I call the ‘soft-core’ fan, the person who just knows those four or five or six songs from the 1980s, they’re not that deeply involved. They come to see me because they used to make out in the car with their wife to ‘Mandolin Rain’ and they’re there for a nostalgic night out. That’s the way of most concerts, if you have created a certain body of work that means something to a sizeable number of people. So I feel his pain and I try to placate him in my own way. For instance, I play ‘Mandolin Rain’ a lot – but I play it in the minor key, because that’s what resonates with me at the moment. And as for ‘The Way It Is’, I really like playing that.”

I suggest that he likes playing Bach’s Goldberg Variations with a bit of “The Way It Is” thrown in the middle: he giggles.

“Look, I am a different person from that old guy from 1986,” he says. “I really am, in every single way, and it is so obvious. My feeling is, I’m trying to give it to you – and I am giving it to you – and I just hope you can meet me halfway. In the end it’s simple. If you really hate it, just don’t come back. You should not come back, because I am not going to be a vehicle for your stroll down memory lane. The people who have their pop moment and spend the rest of their lives replicating that – the people who can do that and really mean it – I admire them. For me, it is a prison and I just refuse. I know I’m asking a lot, especially now that I’m inflicting the modern on them. But if you don’t like it, don’t come. I am fine. I’ve been playing these places a long time. Some people will go. Others are discovering it.”

Hornsby on his “punk instrument” with Jerry Garcia during his time with the Grateful Dead. Photo: Robbi Cohn

At Troy, Hornsby seems to amuse himself most by his experiments in pointillism – the technique of hitting a rapid spray of notes across the keyboard, the musical equivalent of flicking a paintbrush across a canvas. He appears to be tickled by discords and moved by lush melodies, giving a small whoop of pleasure when he hits something he likes. He has called these his “ecstatic” moments. It is something I’ve generally only seen in jazz musicians, that sense of musical surprise, something out of the player’s control. I ask him if every musician gets it.

“I think some people get it because of the amazing arrangements they have created, whereby every night the thing feels transcendent,” he says. “A perfect example is Joe Cocker’s ‘With a Little Help from My Friends’, that incredible version. I get chills just thinking about it. I saw him do it at Woodstock ’94 – I was at the side of the stage and I was just going . . . [He screws up his eyes and gives a little silent scream.] Now, there are not that many transcendent arrangements like that around but most of the greats will have a few of them and I imagine that they’ve created something that will achieve that height every night – it’s just built into the music.”

He appears to get his own thrills out of something as tiny as a chord change. “Or sometimes – sometimes it’s just the sound of the hall. I am an acoustic musician, so I am at the mercy of the instrument and the venue and sometimes the way I played it, the way I struck the chord, the dynamic and the sound of it will just make me go . . . [He shivers.] It’s intangible. And I wish I could do it every night.”

****

Hornsby was raised in Christian Science, the sect founded by New Englander Mary Baker Eddy on the principle that sickness can be healed by prayer. “Let’s see . . .” he says, when I ask him what that meant. “I went to the doctors in second grade when I needed glasses. And I went to the dentist, too. But, sure, there were times when we would have maladies as children and we were able to lose those maladies.”

His mother is a prominent figure in Williamsburg; there is a school named after her. As the church’s community liaison, Lois Hornsby invited patients from the Eastern State mental hospital into the family home; Hornsby and his brothers grew up with people who were, “you could say, way, way out there”. She was also, he adds, “one of the few white hands clapping in applause for integration in this generally conservative town”, though she was not the only family matriarch to campaign against segregation. Hornsby’s aunt, a teacher who served on the York County school board, vetoed the transfer of condemned bleachers from a white school to a black one and was involved in the campaign against the so-called massive resistance movement in the late 1950s, when Senator Harry F Byrd tried to oppose the merging of Virginia’s public schools. Hornsby says that, given his liberal upbringing, he had “no excuse for sucking”.

A friend saw the Grateful Dead at Wembley in 1990 and recalls sonic chaos with “this beautiful, crystalline piano floating over the top”. Three of Hornsby’s keyboard predecessors in the near-mythic band had died in the saddle. “Pigpen” McKernan died of booze; Keith Godchaux, a heroin addict, was killed in a car crash shortly after being ejected from the group. Brent Mydland died of an overdose and Hornsby was brought in to replace him, having struck up a relationship with the Dead when he supported them in the late 1980s. Vince Welnick, who took over from Hornsby when his twins were born, committed suicide in 2006. Hornsby looks bored talking about the rock’n’roll lifestyle (“I always thought it was a little... trite,” he says). He never got into drugs and doesn’t like the taste of alcohol much. He tells me he gets drunk once every two years, on the tour bus, for the amusement of his band but beyond that he has no need. You suspect that his trouble-free lifestyle was an attractive thing for the Dead.

He had seen them aged 18 in the sports hall at the College of William and Mary. They played a five-hour show, then took out the bleachers and did a second night for free in which, he was impressed to see, they didn’t repeat a single song. In one sense, he was a shoo-in for the job because he already knew a lot of the music. But there was a less tangible connection. It is generally acknowledged among Deadheads – the world’s most analytical fan base – that Hornsby’s playing rejuvenated Jerry Garcia’s guitar work at a time when his health was in sharp decline (he died in 1995 after an epic struggle with drug addiction). Hornsby says that on the tour bus, Garcia, whose knowledge of folk and bluegrass was encyclopaedic, in effect gave him a music lesson every day. Then there was the improvising. Onstage, he often had no idea what song the Dead were about to play. He transferred the approach to his own band: The Noisemakers don’t get a set list and are required to read a spontaneous arrangement communicated by his hand signals, or sometimes just the raise of an eyebrow.

In a recent episode of The Looney Tunes Show, Daffy Duck has a piano delivered to his house. “I will fill our home with Beethoven and Rachmaninoff,” he splutters – “and [hands clasped] Bruthe Hornthby. Thhho much Bruthe Hornthby.” The sound of the duck’s flippers on the black notes is not unlike some of Hornsby’s more experimental moments today. It is interesting to think that all over the US, existing “soft cores” might be undergoing a modest, case-by-case conversion to this music, whatever it is. For years he would describe his style as “Bill Evans meets the hymnal” but the co-ordinates are changing. The vague course remains the same: the thousand-seaters of Prescott, Arizona, and Cleveland, Ohio, are his practice rooms. A brief soundcheck to “make friends” with the piano at the venue, a gig, then back in the car and on to the next.

The day after Troy, I made my way to Ridgefield, Connecticut, for the last night of the tour. I had a feeling the menu wouldn’t be the same two nights running. I arrived late at a venue tucked away in the woods in a town of picket fences and SUVs and walked in on the same guy in the same clothes hammering away on a Steinway with the same intensity as the night before. But, sure enough, we got music from the composer Charles Ives, because he was a “Connecticut boy”, and songs that (Hornsby quoted Leon Russell) offered “maximum ‘Shenandoah’ factor”, meaning lush folk tunes – wrung through an east coast intellectual sensibility, a uniquely American synthesis passed from Copland, to Gershwin, to Jarrett, and now this.

The good people of Ridgefield had turned out with children in tow, possibly to hear “The Way It Is”. In the darkness, a woman slipped out to the toilet. When she returned, as his left hand worked away on one of those complicated bits and his right played a flourish over the top, Hornsby cried out, “Tell her what she missed.”

The Grateful Dead play Santa Clara’s Levi’s Stadium on 27 and 28 June, and Chicago’s Soldier’s Field on 3,4 and 5 July

Bruce Hornsby’s next album (working title Rehab Reunion) is released in 2016

Sony release The Essential Bruce Hornsby on 25 May