Descending into Penn Station, it’s not hard to see why New York City’s civic fathers are so embarrassed by the nation’s busiest train station. The station sits bracketed by the Garment District and Chelsea, in the cheapest corner of Midtown, more than half a mile from the gleaming skyscrapers with triple-digit annual per-square-foot rents. The neighborhood (which used to be called the Tenderloin) may be classier than it was when the station was built 100 years ago, but that’s about the only thing that can be said for it.

Penn is clearly subservient to Madison Square Garden, which sits atop it. It was built in the 1960s, at a time when trains were thought to be on their way out, and it shows. The concourse is oppressively low-ceilinged and totally devoid of natural light, and the clear sightlines of the old Penn Station have been replaced with a warren of hallways with an almost airport-like inscrutability. The platforms are narrow and quick to fill up, leading some to call it unsafe—an exaggeration, but not by much.

Penn Station has been a thorn in the side of planners for decades, but in many ways, it’s a fittingly decrepit gateway to a city where politics mire the most minor of railway upgrades.

There are efforts afoot to fix the station, and the lines that run below it. Amtrak is pushing a plan ($800 million) for a new headhouse across Eighth Avenue as well as a new tunnel beneath the Hudson and a full-block station for New Jersey Transit across 31st Street ($15 billion).

Most recently, a drumbeat led by New York Times architecture critic Michael Kimmelman pushed the city to limit Madison Square Garden’s operating permit, with an eye toward eventually moving it and redoing the station underneath. The Municipal Art Society ran with Mr. Kimmelman’s plan, holding a contest for designs; $10 billion, to move the Garden, redo Penn and build parks, was mentioned.

The critic even enlisted Council Speaker Christine Quinn to back the time limit; the City Council set the Garden’s clock at 10 years, at which point, the new Penn backers hope, it will need to move.

Ms. Quinn placed a heavy emphasis on the symbolism of the station. “We need to make sure Penn Station becomes what we need it to be,” she said, “a really 21st-century grand entrance into the greatest city in the world,” not the “bunch of rat tunnels” that it’s been called. (Never mind that during her mayoral campaign, she’s written off even basic subway projects as being too expensive.)

The architects—not, we’d note, engineers or transit planners—invited by the art society ran with the grand visions, focusing almost exclusively on the aesthetics of a new station. Diller Scofidio + Renfro presented a sinuous, aquamarine-tinted jumble of concourses and levels; Skidmore, Owings & Merrill showed a spaceship-like bowl cantilevered over a massive public park. Santiago Calatrava, designer of the PATH boondoggle at the World Trade Center, was even invited to present, though he had the good sense to not participate.

BUT BEHIND the multibillion dollar plans lie a shaky fiscal reality and a bureaucratic morass of railroads and political interests that don’t seem to be up to the task of creating a 21st-century railroad station.

There’s a lot that the three railroads that now share Penn Station can do today to improve what they have, prioritizing cheaper and more useful things over starchitecture and new stations. The simplest is moving Amtrak’s back office space above ground, where there’s more room.

The railroads can merge New Jersey Transit and Long Island Rail Road routes with subway-like through-running service, vastly streamlining operations. They can also pave over redundant tracks to yield wider platforms. They can reorganize the existing station by unifying the railroads’ warring fiefdoms into a simpler, unified broad grand concourse. And over time, a new tunnel can be built beneath the Hudson, emptying into the existing station—or better yet, into a tunnel to Grand Central.

While Penn Station may be busy, it’s far from obvious that its infrastructure is inadequate; in fact it has more tracks than busier stations elsewhere in the world that run more efficiently.

During peak periods, each track at Penn is used, on average, just three times an hour. While Penn Station struggles to move around 300,000 passengers each weekday with more than a dozen commuter tracks, Paris’s main commuter hub, Châtelet-Les Halles, handles half a million with just seven tracks.

The key to making the most out of the existing station is what’s known as through-running. The gold standard in commuter rail integration, through-running would have trains from New Jersey come into Penn Station, but instead of making a capacity-taxing reverse maneuver, they’d run straight out to Queens and Long Island, much like a subway.

As it is now, said Robert Yaro, president of the Regional Plan Association, “we use half the capacity of the station and the tunnels going in and out to service empty trains.”

Amtrak says it’s studying the possibility of through-running before deciding what path to pursue. But it’s hard to miss the fact that Amtrak’s proposed $15 billion overhaul doesn’t even line up the new tracks from New Jersey into Penn with the existing ones to Long Island.

TOO OFTEN, politicians find it easier to cut a check than do the hard work of reforming the intransigent bureaucracies. On the other side of midtown, a $8 billion project under way to bring trains from Long Island directly to Grand Central—East Side Access, as it’s known—would have been the perfect opportunity to eke out a bit more service from Grand Central Terminal’s 67 woefully underused tracks (a world record, in fact). But Metro-North balked at having to share its tracks with its little brother, the LIRR.

So Metropolitan Transportation Authority is currently blasting a $2 billion cavern 15 stories below ground, for the exclusive use of the Long Island Rail Road. Nobody will say it publicly, but from lowly regional planners and former MTA employees to top leaders, most everybody who doesn’t bear direct responsibility for the decision will admit to the error, if only off the record.

And if simply sharing tracks at Grand Central was too tough for the MTA, merging service across state lines at Penn Station would prove nearly impossible. At the very least, it would involve a grueling fight with the various railroad unions (the Long Island Rail Road alone has 11, balkanized into a dizzying array of mid-19th century “crafts,” whose primary purpose appears to protect their work from encroachment by the other unions).

When Philadelphia patched its commuter railroads together bureaucratically in the ’80s after connecting them physically, the city and its suburbs were hit with a crippling strike that lasted more than three months.

And then there’s the management. “If you’re not in charge, you’re fucked” was what the late George Warrington, executive director of New Jersey Transit, told regional planner George Haikalis and other rail activists when they approached him about pursuing a cheaper plan that would rely more heavily on integration.

At the end of the day, through-running would simply involve an interest in transit that no governor since since Nelson Rockefeller has shown.

THE HEADFIRST rush to spend billions while squandering existing assets runs afoul of a basic German transit planning nostrum: organization before electronics before concrete. That is to say, though-running—which involves organizational reform and new electronics—is cheaper than the sort of multibillion dollar “concrete” projects that New York loves so much and should therefore be done first.

This is not to say that large capital projects won’t eventually be necessary, even after the existing service patterns are streamlined. But rather than focus on new station complexes, the railroads should look to a solution considered for the first Hudson River crossing that Governor Chris Christie ultimately killed: a tunnel between Penn Station and Grand Central.

New Jersey Transit and Amtrak need a new tunnel beneath the Hudson, but they also need somewhere to put the trains once they reach Penn. A tunnel from Penn Station to Grand Central would achieve this while also allowing for through-running not only between New Jersey Transit and the Long Island Rail Road, but also between Metro-North and job-rich suburbs like White Plains and Stamford, Conn. Not to mention giving Metro-North riders a one-seat ride to Penn Station, and New Jersey Transit riders a similar trip to Grand Central.

And not only would it be more useful, but it would also be cheaper—a report done by New Jersey Transit, the MTA and the Port Authority found the construction and operating costs to be less than the alternative of stub-ending the trains at a new station near Penn (a version of what Amtrak now wants to do at Penn Station South).

The fact that the cost of digging a tunnel beneath one of the densest parts of New York would be on par with that of building a new station might seem odd. But anyone who’s seen the astronomical cost overruns in station caverns for East Side Access and on the Second Avenue Subway (where for every dollar spent on tunnels, at least three are going toward stations) won’t be surprised.

New York’s transit agencies have not only shown no interest in this plan, they have also fought aggressively to prevent the release of the study that outlined it.

NEARLY EVERY part of Penn is inadequate, and it’s not hard to see, in a better world, plans befitting the busiest station in America.

A century ago, the Pennsylvania Railroad spent a considerable sum of money on the original Pennsylvania Station, hiring the best architects and making an enormous gamble on untested tunneling technology. But the old Pennsylvania Railroad was the gold standard in American railroading, a shining beacon of efficiency and good management—the polar opposite of Amtrak, New Jersey Transit and the MTA today.

New York City has a way of holding out for grand transit plans rather than pursuing more realistic ones. Instead of sitting around and waiting for the feds and real estate developers to cut 11-digit checks to remake blocks of the city, the governors of New York and New Jersey should knock heads together to make cheaper short- and medium-term improvements their highest priority.

Only once the railroads have shown that they can cooperate and work toward a modern, unified and efficient system, does Penn Station merit the grand entrance—the crowning achievement of a 21st-century rail network, not an end unto itself—that its politicians and civic fathers have been dreaming about.