High up on a hill overlooking the Harlem River lies not only Manhattan’s oldest home, but one of its supposedly most haunted, the stately Morris-Jumel mansion.

The mansion was originally built in 1765 as the home of former British Army officer Roger Morris. When Morris fled America for England at the outbreak of the Revolutionary War, the house was confiscated along with other Loyalist properties by the Commissioners of Forfeiture. Following his defeat at the battle of Long Island in 1776, George Washington took refuge and set up his headquarters here, at the highest point in Manhattan.

The house itself is a glorious example of preserved Palladian architecture. Its immaculate white framework is prominently on display atop Coogan’s Bluff, but it rests fairly anonymously way up in the Washington Heights neighborhood of Upper Manhattan, far from where most New York City tourists and visitors wander.

After serving time as a popular tavern on the old Albany Post Road, the mansion was purchased by wealthy French merchant Stephen Jumel in 1810. It’s at this point in the venerable house’s history that tales of the macabre begin. Jumel brought with him his mistress, and then wife, Eliza Bowen Jumel. He remodeled the house in the popular Federal style of the time, adding a columned porch, and redecorating the interiors. At one point Stephen and Eliza were among the wealthiest families in Manhattan. They planted vineyards of Bordeaux grapes that still run wild today in neighboring Highridge Park.

When they traveled to Paris, the Jumels were acquaintances of Napoleon and Josephine Bonaparte. But in New York, they were shunned by high society. Eliza came from a working-class Catholic family in Rhode Island, and cruel rumors circulated that she had worked as a child prostitute in her mother’s brothel. When Stephen Jumel died in 1832, the exact cause was unknown. Some reports had him dying from pneumonia, or from injuries sustained in a carriage accident. Others had him passing away after falling on a pitchfork. The gossip of the ladies’ parlors in Manhattan was that Eliza had opened up his dressings after the pitchfork incident and watched him bleed to death. A local medium claimed that Eliza had buried her husband alive.

Regardless of how the unfortunate Stephen actually passed away, what is certain is that only months later Eliza took a new suitor: Aaron Burr, the highly controversial former Vice President of the United States who had notoriously killed Founding Father Alexander Hamilton in a duel in 1804. Despite Burr’s age (he was 77), they soon married in the front parlor of the mansion. The marriage was short-lived, however, and Burr died three years later on the exact day the divorce was granted. Eliza took to calling herself the “widow of the ex-Vice President of the United States.” Now fabulously wealthy, Eliza lived on in the giant house as a recluse, gradually falling into dementia. She was said to have walked the hallways and chambers of her mansion, her hair unkempt and her clothes soiled, haunted by the tortured souls of her past lovers. She died alone in the house in 1865 at the age of 90.

The city of New York eventually purchased the mansion in 1904, and it was added to the National Register of Historic Places. Today, it’s operated as a museum with period-decorated rooms open to the public, an unusual old stately home tucked away in the hills of Washington Heights. Believers in the paranormal claim to have seen at least five ghosts inside: a serving girl, a British soldier from the Revolutionary War, the doomed Stephen Jumel, the figure of Aaron Burr, and most prominently, the figure of an elderly woman in a violet dress, the troubled old mistress of the house, Eliza Bowen Jumel.