Things might have been different if only Jack Kerouac had known how to hop a freight train. His attempts, described at the outset of the book that gained him celebrity, were a failure; a romantic notion from a bygone era that, in reality, proved tricky. Much easier to hitch a lorry ride. And thus the writer’s euphoric westward peregrinations, by motor car, kindled a new spirit of domestic US travel: On the Road rather than On the Railroad.

Since the 1950s, America’s once great “iron horse” has suffered under the exhaust fumes of the four-wheeler. In recent years, however, national Amtrak services have seen an increase in passenger numbers, not just in the north-east’s multi-city corridor but on long-haul routes, too. Investing millions in improving customer service, safety and, crucially, sustainability, the railroad may be on track to regain its former glory.

I am thinking of Kerouac as our train, the Southwest Chief, trundles over a moonlit Mississippi River, exchanging cornfield Iowa for the western plains, as he did more than 70 years ago. My brother and I are travelling from Illinois to Arizona – to see the Grand Canyon – aboard one of Amtrak’s long-distance Superliner services. Having left Chicago’s art deco Union station shortly before 3pm, a 30-hour journey will swing us through the states of Missouri, Kansas, Colorado and New Mexico before the Chief chugs on to Los Angeles.

To fly to the canyon would have taken roughly a quarter of the time, on a ticket costing a third of the price. It would have freed up hours to hike more of the Colorado River valley and to sink beers in the craft breweries of Flagstaff. Alternatively, to drive would have meant Route 66 – the iconic American road trip into the west. So why take the train?

“Climate guilt,” as fellow passenger Michael puts it. Increasingly uncomfortable flying to attend interstate conferences, the academic from New Mexico is returning from Illinois on the Southwest Chief. “I had to convince my university to cover the extra day’s travel, plus a sleeper ticket,” he tells me in the dining car. “Fortunately, they were understanding.”

Facebook Twitter Pinterest Chicago’s Union station. Photograph: Tom Collins

Reports for the US Department of Energy show that, per passenger mile, rail has a lower energy footprint than road and air (despite the diesel locomotives). And with Amtrak declaring a 17% reduction in greenhouse gas emissions since 2010 – and more efficient, high-speed engines on the horizon – Michael, like me, is in favour of supporting rail. But I was curious, too. At some stage, all my American road trips have intercepted a train ploughing off into deep country; I have so often wanted to be aboard. And now, with the lights of Kansas City disappearing into the night, and Elvis’s Mystery Train loud in the headphones, at last I am.

Morning, and dawn casts an orange glow across wintry Kansas, flaring the curtains of my lower bunk window. My brother and I are sharing a “roomette” – a cabin comprising two seats that convert into bunkbeds – and he drew the shorter straw. Though arguably just as comfortable, the top lacks a window view. Miles of prairie are lighting up from the east, studded by cattle and deer: interior America at its biggest.

The engine issues a series of whistles and hurtles over a crossroads. We rouse ourselves with coffee from a round-the-clock dispenser in the sleeper gangway and sway along to the sightseer lounge. Here, the sunrise can be enjoyed via high curving windows and outward-facing seats. We join a handful of passengers: middle-aged Debby, who’s been quietly “watching all the back yards of America” since Missouri; a pair of jet-lagged Australian retirees en route to LA, and an Amish lumberman from Ohio, off to Colorado for “elk season” hunting. The Amish are frequent patrons of the Southwest Chief, as are Mennonites from rural midwestern communities.

Facebook Twitter Pinterest Sightseer lounge on the Southwest Chief. Photograph: Matt Collins

Debby has been to visit her daughter in Chicago. “Did you notice how simple it all was?” she asks, when I remark on yesterday’s smooth departure. “You show your ticket, and off you go; nothing like an airport.” Breakfast is announced over the PA and we move to the adjoining dining car. A restaurant attendant seats us at a four-berth table opposite Linda – Californian, mid 70s – a regular on this route. As Kansas whirls by over eggs and fried potatoes, we discuss her fear of flying: “Trains are the only way to fly,” she giggles, “but you’ve got to have a sense of humour.”

Humour is required for the hold-ups. As private freight operators own these tracks, Amtrak’s government-subsidised passenger services must give way to passing cargo, occasionally causing delays. Weather, too, can play with the timetable: in 2014 heavy snowfall stranded three Amtrak trains overnight in rural Illinois. Disruptions like these can discourage business travel, “That’s not why I take the train,” says Linda. “I enjoy the interactions, meeting people – and the views you just can’t get on the freeway.”

Facebook Twitter Pinterest View from near La Junta, Colorado. Photograph: Matt Collins

These elements – social and visual – combine to offer an experience unique to rail. Before lunch, my brother and I are chatting with our respective neighbours, aperitifs in hand, as the Chief climbs the Raton Pass from Colorado into New Mexico along a slip of old Santa Fe railway. A glorious view of dramatic rock, golden grass and snow-clad juniper promotes easy conversation. The nature of the seating, too – particularly in the dining car where tables are shared – is conducive to making introductions. Talk isn’t mandatory. It is simply that, unlike the isolation of a car or the buttoned-up nature of planes, American train travel is wonderfully social, from the downstairs cafe car and upper lounge to the short stops at which passengers disembark for a smoke, a communal leg stretch or, as at Albuquerque, a chat with Navajo stallholders on the platform.

But the historic views are the main attraction: Mark Twain’s Mississippi, Jessie James’s Missouri hide-outs; the New Mexican stomping grounds of the Apache and Navajo nations; and the distant white-capped Rockies. “See the indent in the ground,” points solo traveller John excitedly, “that’s a stretch of 19th-century wagon trail.”

Facebook Twitter Pinterest View from the train near Raton, New Mexico. Photograph: Tom Collins

Superliner routes such as this one include four sleeper-car options besides standard coach (other trips include LA to New Orleans on the Sunset Limited and Chicago to Seattle on the Empire Builder). Our two-berth roomette is the basic choice with its folding table, temperature control, plug sockets and access to the carriage shower. Next, there’s a three-person en suite bedroom (the lower bunk accommodates two); a “family bedroom” (four beds) and a wheelchair-accessible, two to three-person en suite. These rooms may lack the opulence of the Southwest Chief’s predecessor, the “Super Chief” – whose lavish veneered interiors charmed LA-bound Hollywood stars in the 1940s – but they are quiet and comfortable. A conscientious attendant accompanies the sleeper car, prepping beds and taking dining car reservations. At night, the carriage can rock while negotiating a bend and freights enter your dreams as their cargo thrums past but these minor intrusions authenticate a true railway experience.

At around 9.30pm our attendant announces Flagstaff. Stepping out, we greet the dry, chill air of Arizona’s high desert and head to the Weatherford Hotel. This three-storey downtown relic with wraparound balconies and period pillars is rich in western history. Inside, turn-of-the-century furnishings and the aroma of woodsmoke and whiskey recall its lively heritage, as do the ghosts that are said to haunt the hallways and rattle the pipes at night.

Facebook Twitter Pinterest Train going through Flagstaff station. Photograph: Alamy

Similar to the Weatherford, the railroad arrived in Flagstaff in the late 19th century, bringing jobs, export connections and prosperity. This was shortly after the completion of America’s first transcontinental railway, an ambitious feat that commenced while the country remained at civil war. “The result,” wrote railroad biographer Christian Wolmar, “was to enable America to be governable as a single nation ... a single country stretching from coast to coast.” It’s hard to imagine how such pivotal infrastructure yielded over the next century to Henry Ford and the open road; particularly here, where the tracks pass directly through town.

Our short stay in Flagstaff is appropriately Arizonan: we photograph sunsets, devour Tex-Mex and marvel at the Grand Canyon, with its puzzling depth and winding trails occupied by mountain goats and blue Mexican jays. My brother does the pose – perched on a precipice – while my nerves get the better of me. At Kaibab national forest we take in swathes of ponderosa pine, smoky with the Forest Service’s controlled burns, and the evenings pass with pale ales in Beaver Street Brewery.

Facebook Twitter Pinterest Grand Canyon view. Photograph: Matt Collins

Days later, at 5am, we are back on the platform awaiting the eastbound to Chicago (the Southwest Chief runs only once a day, so departure hours can be antisocial). Tracing back over all that sage and desert, through icy Colorado and into the bright midwest, we revisit the small towns for whom this service is a transportation lifeline; Lamy, New Mexico; Newton, Kansas; La Plata, Missouri. Conversations onboard convey concerns for these outlying communities, as Amtrak endeavours to reduce its high overheads. “‘This train connects us to the world,” a Colorado local tells me, adding “I worry for its future.”

Recently, the Southwest Chief narrowly escaped a proposed replacement bus service between Albuquerque and Dodge City, where, since BNSF Railway rerouted its freight, Amtrak has been footing the line maintenance. Efforts to modernise are proving similarly contentious, the traditional dining car experience – steak and cheesecake served on white linen – is being replaced on some eastern routes with prepackaged meals deemed more appealing to millennial riders.

Modernising will be necessary if Amtrak is to entice drivers from their cars and continue its greener, low-emission trajectory. Nonetheless, an element of tradition is no doubt part of the allure of cross-country vacation routes such as this one: a hard balance to strike. My own experience on the Chief, however, was an overwhelmingly positive one, and full of memorable encounters. Like the young man who took our picture as we chugged back into Illinois, just married in Vegas and still on a high. He and his wife made the round trip by rail – something old and something new.

• The trip was provided by Amtrak. A return ticket with Roomette accommodation on the Southwest Chief costs $1,122, including all meals. Accommodation was provided by Discover Flagstaff at the Weatherford Hotel (doubles from $115, room-only)

Looking for a holiday with a difference? Browse Guardian Holidays to see a range of fantastic trips