Steam engines were introduced fairly early, but occasional boiler explosions provoked attempts to eliminate locomotives in built-up areas, which the railroad successfully resisted for years. A midline depot stood at Fourth Avenue and 26th Street as early as 1845, when it burned, and in 1847 The Evening Post reported a “spacious freight house” at that site. It is possible that was the picturesque castlelike building seen in the 1860s view above left, at the northwest corner of 26th and Fourth, stretching back to Madison Avenue.

If so, it might have been designed by Robert G. Hatfield, who did other work for the New York and Harlem around that time.

In 1857 a separate but related line, the New York and New Haven, erected an adjacent structure, at the southwest corner of 27th and Fourth, an awkward Italianate building, which like its neighbor stretched back to Madison Avenue. The rail lines turned into a yard shared by the two terminals, and this was the beginning of the Grand Central idea — different rail companies would use the same, centralized facility. Ultimately, Grand Central Terminal would serve the Hudson, Harlem and New Haven lines.

In 1858, a case of not-in-my-backyard syndrome erupted on Fourth when some rich residents of town houses in newly fashionable Murray Hill got the Common Council to consider banning steam south of 42nd Street. But a group of commuters met at the station to protest switching back to the much slower horse-drawn cars.