THEY missed something and it is a rather big something. When the Pennsylvania Railroad demolished Penn Station in 1964, it was described by The New York Times ''a monumental act of vandalism.'' Many New Yorkers and architectural historians still regret the loss of the station, and aficionados search the lower depths of the present station for remnants of the old one - handrails, elevator doors, lighting fixtures and the like.

But there is one little-noted element of the old station left, a monumental building in its own right, which is itself vulnerable to demolition. It is the grimy 1908 Penn Station Service Building at 242 West 31st Street, across the street from the present station.

As the 19th century gave way to the 20th, the Pennsylvania Railroad was chafing under a peculiar burden - it had no station in New York City. Passengers had to disembark in Jersey City, crossing the Hudson to Manhattan on ferries. It was an embarrassingly awkward entrance to the nation's greatest city for one of the nation's greatest railroads, and it lost passengers bound for destinations like Chicago, who could go by a competing line, the New York Central, from Grand Central Terminal.

Tunneling under the Hudson was less expensive than a bridge, but steam locomotives could not regularly negotiate the length of the tunnel. By the turn of the century electric engines, like those on the new IRT subway designed in 1900, were being developed, even for the heavier intercity trains.