The first Cliff House was built in 1858, above Ocean Beach, in west San Francisco. It has been rebuilt five times since for various reasons, such as remodeling or damage.

In 1894, the third, and most photographed, incarnation of the house was built by Adolph Sutro, a successful mining engineer. Sutro built the seven-story mansion in Victorian style, an elaborately decorated structure dubbed the "Gingerbread House."

Cliff House was the scene of a number of historic events, including several shipwrecks. A wreck in 1887 caused damage to the second Cliff House when the dynamite on the ship exploded. The first ship-to-shore transmission, using Morse Code, was received here in 1899 and in 1905; the first radio voice transmission was sent from the house to a point a mile and a half away.

Cliff House survived the earthquake that struck San Francisco in 1906 with only minor damage. It burned to the ground the following year, however. Sutro's daughter began the construction of a new Cliff House restaurant in 1908, but on a vastly smaller scale. And so it is today.