Frederick Koch’s Upper East Side townhouse sits on a quiet street off Fifth Avenue, near the Metropolitan Museum of Art. It is January 2013, and in a little over a week’s time, Frederick’s hard-hat-wearing younger brother will dip a ceremonial shovel into a mound of dirt in front of the museum, officially breaking ground on the David H. Koch Plaza. Frederick, the eldest of the four Koch brothers, has devoted his life to the arts, but it’s David’s name that’s plastered on some of New York’s most prestigious cultural real estate, including the former New York State Theater at Lincoln Center.

The world of the Upper East Side elite is a small one, and Frederick and David occasionally bump into each other at galas or charity functions. These short, awkward exchanges (“Oh, hi, Freddie”) are pretty much the extent of their contact. Unlike David, who enjoys the status that comes with his high-profile philanthropy, Frederick conducts his life as if almost striving for obscurity. Thanks to the recent infamy achieved by David and Charles Koch through their sprawling political operation, he is now thought of as one of the “other” Koch brothers (along with another brother, Bill).

Frederick, 80, is so private about his affairs that during the 1980s, after underwriting the $2.8 million construction of England’s Swan Theatre in Stratford-upon-Avon (in the shell of a fire-ravaged Victorian playhouse), he kept quiet about his gift for several years, as the British press tried to dig up the name of the angel donor. When Frederick’s role was finally revealed, he told the BBC in a rare interview, “Never ask from where I came, nor what is my rank or name.” He was quoting Lohengrin’s warning to Elsa, when the knight comes to her aid in Wagner’s romantic opera. When Elsa later poses the forbidden question, her savior disappears in a boat pulled by a dove.

Built of white marble, Frederick’s seven-story neoclassical townhouse is one of a trio commissioned, in the early 1900s, by dime-store magnate Frank Winfield Woolworth, and designed by Charles Pierrepont Henry Gilbert, one of several architects favored by New York’s industrialists during the Gilded Age. Woolworth gave the townhouses to each of his three daughters as wedding gifts. Six East 80th Street, the property Frederick now owns, belonged to the tycoon’s youngest daughter, Jessie. Woolworth was said to have wept on the day he gave Jessie away to James Donahue, the Irish-American scion of a less prominent family that made its money in the fat-rendering business. Donahue was a man of vices—gambling, booze, and young men, in particular—and Woolworth’s doubts about the pairing proved well placed. In 1931, during a luncheon at the couple’s home, the 44-year-old Donahue excused himself from the table and locked himself in the downstairs bathroom. He staggered out a few minutes later exclaiming, “I’ve done it.” Despondent over his finances and recently spurned by a young sailor, Donahue had gulped down seven mercury-bichloride pills. He died within days.

Frederick acquired the property for $5 million in 1986, three years after cashing in his stock in the family company, Koch Industries, and in the throes of a frenetic buying spree of historic homes and artwork. Frederick’s longtime architect, Charles T. Young, spent the next decade restoring the townhouse to its former splendor, and in many cases, surpassing it. Frederick spared no expense. He continued a marble balustrade that had ended after the first floor up the staircase and through the remaining six floors of the house. He replaced the crumbling plaster walls with limestone from the quarries of Caen in northwestern France, the kind that would be found in a Parisian townhouse of the French Régence period. In one case, Frederick made structural alterations just to create enough wall space to hang one of the masterpieces of his art collection, widening a pair of stone columns to fit William-Adolphe Bouguereau’s The Abduction of Psyche.