His sons, Paul and Seymour, grew the flooring company into a real estate and construction empire, but open warfare broke out in the early 2000s among the next generation. One faction was led by Paul’s son Howard Milstein; the other by Seymour’s son, Philip L. Milstein, who is Larry and Toby’s father.

“It was one of the more legendary family squabbles in the history of New York City real estate,” said Amir Korangy, the publisher of The Real Deal, a real estate trade magazine that covered the legal battle between the cousins. “There was a $5 billion family fortune at stake, and it tore the family in two.”

After the schism, the companies Milstein Properties and the Emigrant Bank went to Howard, while Philip and his sister, Constance Milstein, founded Ogden CAP Properties, which owns One Lincoln Plaza and other prime real estate.

Such wealth allows Larry and Toby to control personal foundations that disburse up to $1 million each, every year. Beneficiaries include such uncontroversial groups as Charity: Water, New Museum and the Lincoln Center for the Performing Arts, as well as more socially minded causes like the Coalition for the Homeless and anti-gun-violence campaigns.

For some groups, the Milsteins open their home, and it was a gathering for an arts charity, A BroaderWay, that inspired them to try a musical method for connecting with the house spirits.

“As young adults celebrating the arts, what better way to pay respect to the presence of the history here than singing a few notes from their favorite songs?” Mr. Milstein said. “It’s a modern séance that’s rooted in the music of the individuals that inhabited this place. You can feel the vibrations of them, and that means they’re there.”