In the early 1600s, Johann Caspar Richter moved from Denmark to the town of Kronenschieldt in Upper Saxony (present Germany) and married Maria Hahn.

The family then added the town's name to their own. Their son Johannes Caspar Richter von Kronenschieldt, after being educated in Leipzig, moved to Boston around 1688. Arriving in this English-speaking area, he anglicized his name to John Caspar Crowninshield. In 1694 he married Elizabeth Allen, the daughter of a ship captain. They lived near Spring Pond that was the border between Lynn and Salem. Little did they realize that they were the beginnings of a family dynasty that would stretch through generations of power and influence in Salem and beyond. Their family would be included in the listing of America's most powerful families and be considered one of the original Boston Brahmins.

The first native born generation sought and found their livelihood and growing fortune pursuing life in the growing port of Salem. John Crowninshield, achieving some success as a fish merchant and ship captain, built the above modest house at 106 Essex St. It was here that four generations of the Crowninshield Family lived. As the family fortune grew throughout the 18th century from the spice and far east trade as well as privateering during the Revolution, other mansions were built, but this house remained in the family. The Crowninshield family, thanks to their trading expertise, became one of the wealthiest families of Salem. George Crowninshield (1733-1815) founded the Crowninshield & Sons Shipping Company with his five sons. They built the Crowninshield Wharf, (where the power plant is now) that serviced the far east trade. It was Captain Jacob Crowninshield, one of George's sons, who purchased in India the first elephant brought to America. When he did so he wrote that he had used his own money, $450, to make the purchase and if it was a folly it was his alone. He later sold the elephant in New York for $10,000. The elephant was a spectacular hit as it toured the country.

While there was competition and some antipathy, especially regarding politics, among Salem's elite, wealthy families, they shared the same society. That closeness produced a number of intermarriages. The Crowninshield family intermarried with the Derby, Endicott, Gardner, Peabody and Pickman families over the years. As shipping waned, successive generations became involved in government, the arts, and a number of charitable enterprises. Over the generations, the family has produced a number of state representatives, senators, congressmen and cabinet members under Presidents Jefferson, Monroe, Madison, Cleveland and Hoover. There were also a number of Generals and Admirals in the armed forces. In the arts, the family has produced notable painters and craftsmen.

Reverend William Bentley, the pastor of the East Church and noted Diarist of Salem during the East India Trade, boarded with the Crowninshield family from 1791 to his sudden death in 1819. At this time Hannah, the widow of Jacob Crowninshield, to make ends meet, was taking in boarders in her half of the house that had been divided at John's death. The other half of the house was owned by her son, Benjamin, who, as a ship captain and merchant, was very successful. His half of the house was expanded and rennovated during his tenure.

Rev. Bentley's 11-volume, (abridged), diary is still a popular source of everyday knowledge of the town of Salem and its people at the height of the East India trade. He was a noted scholar and frequent correspondent to Thomas Jefferson, often translating and advising. Jefferson tried to recruit him as the first president of University of Virginia as well as Chaplain to the Congress. Bentley declined both offers in favor of staying in his much loved Salem. His room was on the right hand end of the second floor. When the house was on Essex Street, his view looked down Union Street toward the old bridge.

By mid 19th century, most of the Crowninshield family had moved from Salem to Boston as well as New Hampshire, Vermont and New York where they became involved in local and state affairs and are often commemorated by streets named after the family. Over the next several decades the house changed hands. Through most of the 19th century, the house remained a residence nestled behind the Franklin Building that housed the East India Marine Society that the Crowninshields helped to found. In 1924, The Franklin Building was replaced by the Hawthorne Hotel.