Wall Street is arguably the most famous street in America. Just as the word “Broadway” has become synonymous with American theater—despite 99 percent of the street having nothing to do with acting—“Wall Street” is now shorthand for the entire financial system. This symbolic weight is so heavy that it has buried the thoroughfare’s long and colorful history. In fact, many visitors (and a few locals) are surprised to learn that the street’s name memorializes the wooden wall that once ran across Manhattan.

Wall Street is less than half a mile from end to end, but contains within its short blocks a microcosm of the history of the city. Once a promenade that took people past some of Manhattan’s most elegant homes, the street has played host to commercial piers, coffee houses, the city’s slave market, and the U.S. capitol building where Washington was inaugurated and the Bill of Rights ratified. It has been home to an array of private banks, the U.S. Custom House, the assay office, and, of course, the New York Stock Exchange, which originally met outdoors, under a tree.

If the Downtown Alliance gets its way, Wall Street is again on the verge of upheaval, and the changes instituted after the 2001 World Trade Center attack will be reworked into a truly pedestrian-friendly enclave. With the boom in downtown real estate and the conversion of One Wall Street into condos likely to fetch $2,000-$3,000 per square foot, Wall Street is poised to become, once again, one of the city’s most attractive residential areas. But will this new Wall Street be an improvement? While making any city more walkable is an underlying principle of most contemporary urban design, will a pedestrianized Wall Street—already clogged every day with tourists and brokers—become more like Times Square?

New Amsterdam, Manhattan’s first European settlement, was built over the course of three decades, fanning out from what today is the intersection of Pearl and State streets. In 1625, Fort Amsterdam was erected where the Museum of the American Indian now stands. Bowling Green was a market square and Pearl Street, running along what was then the East River shoreline, was home to the town’s first church, the city tavern—later converted to City Hall—and the homes of many of the settlement’s most prominent citizens.

In May 1652, the first Anglo-Dutch War began. Though the war would ultimately be confined entirely to European naval battles, the citizens of New Amsterdam decided to erect a defensive palisade in case English colonists in America attempted an overland attack of the Dutch. The public paid for it, at a cost of 5,050 guilders. (The entire purchase of Manhattan 25 years earlier had cost just 60 guilders.)

The 9-foot-high wall went up quickly, built by enslaved people and volunteers. It ran from the Pearl Street edge of the island west to the Hudson River shore, which was then about the level of today’s Church Street, behind the Trinity Church cemetery. The defensive line was 2,340 feet long, built in 15-foot-long sections.

Director General Peter Stuyvesant and the town council ordered that “burghers of this City shall stand guard in full squads over night at places,” which meant providing a pathway south of the new palisade that would accommodate the settlement’s recently formed Burgher Guard: literally, a wall street. The Dutch called the palisade “Het Cingel” (the belt), but there’s no concrete evidence that they had any particular name for the street.

The wall was first put to the test in 1655. With Stuyvesant and most of the city’s able-bodied citizens on a quasi-military expedition to Delaware—where the Dutch felt the Swedish were encroaching on their territory—the city was attacked by American Indians. The so-called “Peach Tree War” began when an American Indian woman was caught stealing fruit from the orchard of Hendrick van Dyck, whose farm faced the Hudson River just below the wall. Van Dyck shot and killed the woman; soon, hundreds of American Indians amassed outside the wall and simply walked around the defensive line. After sacking New Amsterdam, taking hostages in Pavonia (now Jersey City), and burning fields in Staten Island, the American Indians retreated, and when Stuyvesant returned from Delaware, he had to negotiate an arms-for-hostages trade using gunpowder to buy back the colonists.

In the wake of this attack, the wall was strengthened, with a Hudson River flank added. It was also raised higher, and the nascent Wall Street widened as a military parade, creating the basic layout of today’s thoroughfare. (My tenth-great-grandfather, Johannes Nevius, who would later become the secretary of New Amsterdam, forfeited his prime property on the southeast corner of Broadway and Wall Street so that the street widening could happen.)

When the English showed up in 1664 to conquer New Amsterdam, the wall was no deterrent. In fact, the English didn’t have to fire a shot. Outgunned, the Dutch surrendered the city intact, and overnight New Amsterdam became New York. The new English rulers kept the wall for a generation, paying exorbitant sums for its upkeep (one observer called it a “monument to our folly”), until finally deciding to tear it down in 1699 to make room for, among other things, a new City Hall at the corner of Wall and Nassau streets, completed in 1703.

Meanwhile, the island itself was expanding. As landfill was added, the East River shoreline was pushed from Pearl to the aptly named Water Street, and then later to Front Street, with Wall Street expanding with the island. In the early 18th century, the border would be pushed again to South Street.

In 1694, the first wharf was built at the foot of the street at the East River, and the eastern end of the street began to grow into a mercantile center. New York was also a hub for piracy in the colonies (which helped our port compete with Boston and Philadelphia), and many of New York’s elite—such as Captain Kidd—found profit in plunder and privateering.

At this new mercantile end of Wall Street, the city’s slave market was established circa 1709-1711, “at which place all negro and Indian slaves to let out to hire, or be sold, took their stand.” Throughout the 18th century, around 40 percent of white households in the city owned slaves. The market took on other functions, including selling grain, and was dubbed the Meal Market, though slaves continued to be sold there until 1762, when the market finally closed. (Slavery itself continued to be legal in New York until 1827, and in some cases people remained in bondage even longer.)

Wall Street is less than half a mile from end to end, but contains within its short blocks a microcosm of the history of the city.

While the end of the market may have been precipitated by a downturn in slaveholding in New York, it’s more likely that its dissolution was prompted by the changing character of the street. Though a pier remained at the East River foot of the street, the 1767 Ratzer Map shows most of the docks were now in the area north of Wall Street.

Instead, Wall Street was becoming an upscale address. At the time the controversial Stamp Act was passed in 1765, marking the beginnings of the American Revolution in the city, the newly appointed stamp collector for the city, James McEvers, lived “in an elegant new mansion in Wall Street, corner of William.” Similarly, Samuel Verplanck, head of one of New York’s most famous families, had erected a “lavish, 40-foot-wide home” that abutted City Hall.

At the Broadway end of the street stood the first Anglican parish, Trinity Church, built in 1698 with financial help from Captain Kidd. A few doors away on Wall Street was the First Presbyterian Church, and the Dutch Church, formerly located inside Fort Amsterdam, was one block away, on Garden Street (today’s Exchange Place). The 1703 City Hall at the corner of Wall and Nassau streets was a hotbed of political activity, including the trial of John Peter Zenger, whose acquittal established truth as a defense against libel, and the Stamp Act Congress of 1765, where representatives of all 13 colonies met to strongly protest parliament’s imposition of taxes. It may have been here that the words “no taxation without representation” were popularized.

In 1770, when the golden equestrian statue of King George III was added to Bowling Green, a companion statue of William Pitt was put up at the corner of Wall and William—pointedly in front of the home of stamp collector McEvers—as a “public testimony of the grateful sense the colony of New York retains of the many eminent services he rendered America, particularly in promoting the repeal of the Stamp Act.”

However, the repeal of the Stamp Act did little to quell the colonies’ revolutionary fervor, and when the war broke out, Manhattan became highly prized. On September 15, 1776, the British chased George Washington and his troops out of Lower Manhattan; five days later, a fire broke out in a tavern near the Battery, quickly spreading up the west side of the island toward Wall Street. (The question of whether the fire was an accident or arson remains unanswered.) Though a few blocks south of Wall Street burned, the fire was mostly contained to the west side of Broadway. Trinity Church was lost—along with approximately 500 to 600 other buildings—but everything else on Wall Street remained intact.

The Presbyterian Church, whose members were staunch patriots, was requisitioned as a British barracks; the nearby Baptist Church became a stable. City Hall, meanwhile, became a headquarters for the Redcoats. At the time of the fire, much of Wall Street would have been abandoned: It’s estimated that as much as 80 percent of the city evacuated at the outbreak of the war.

Very little of what burned down in the fire was cleared or rebuilt during the war, and the British may have used the ruins of Trinity Church as the backdrop for a summer theater.

Though the British surrendered at Yorktown in 1781, the war didn’t come to an end until November 25, 1783, when the British evacuated New York. George Washington led a parade of military and civilians “down Chatham street, passing a body of the British troops which were still halting in the fields (now the City Hall Park) ... down Pearl Street, proceeded on to Wall Street, and up Wall, then the seat of fashionable residences, to Broadway….”

On this triumphal entry to the city, Washington would have passed the old City Hall, little realizing that in six years it would become his office as the capitol building.

As the Articles of Confederation began to fray in 1785, Congress decided to move the nation’s capital to New York, in part because Philadelphia was not seen as neutral ground. Following the Constitutional Convention and the selection of Washington as the first president, New York’s City Hall was remodeled by Pierre L’Enfant and reborn as Federal Hall, and it remained the seat of government until August 1790. It was here the Bill of Rights was ratified and numerous early laws passed by the first Congress.

When New York was the capital, it was also the center of American finance, and Wall Street soon became home to the city’s burgeoning securities market. Traders began to buy and sell private shares of stock and loans in the form of bonds. As Charles R. Geisst notes in Wall Street: A History:

As the nascent securities business quickly grew, the traders divided themselves into two classes—auctioneers and dealers. Auctioneers set the process, while dealers traded among themselves and with the auctioneers. This early form of trading set a precedent that would become embedded in American market practice for the next two hundred years. The only problem was that the auctioneers were in the habit of rigging the price of the securities.

At Treasury Secretary Alexander Hamilton’s urging, a group of traders regularized the market by signing the Buttonwood Agreement on May 17, 1792, which allowed their select group to sell stocks and government bonds while avoiding inflationary auctions. The name came from the tree under which traders would meet, outside what is today 68 Wall Street; non-members could also come trade, but at higher prices. This—along with the exchange’s $25 membership fee—was discouraging, and a lively curb market blossomed on Wall Street for those for whom the Buttonwood Exchange was too exclusive. This outdoor exchange would persist in one form or another until 1921.

Beginning in 1796, New York’s only bank, the aptly named Bank of New York, was also located on Wall Street. Founded in 1784 by Hamilton and his associates, the bank had an exclusive charter, which meant that the only other bank operating in the city was the branch of the federally chartered First Bank of the United States, which had its offices in the former Verplanck mansion on Wall Street.

Together these three Hamiltonian institutions—the Exchange, Bank of New York, and Bank of the United States—cemented Wall Street’s financial dominance in the early Republic. Soon after the founding of the exchange in 1792, the Buttonwood traders moved indoors to a Wall Street coffee house they constructed called Tontine’s. They remained there until 1827, when a new Merchant’s Exchange opened on Wall Street near William Street. Inside, the members of the exchange erected a marble bust of Alexander Hamilton.

With the opening of the Erie Canal in 1825, New York secured its place as the country’s main port, bringing the city even more money. Unsurprisingly, Wall Street banks proliferated. The first competition to the Bank of New York had come in 1799, when Aaron Burr circumvented the Bank of New York’s charter and created the Bank of the Manhattan Company. Soon the floodgates opened; by the time the Erie Canal was finished, the state of New York had 34 banks. Ten years later, there were 86, many connected, directly or indirectly, to the shipping industry.

No trace of Wall Street in this era remains.

In 1835, much of Lower Manhattan was destroyed in New York’s biggest fire, a conflagration that wiped out 700 buildings and was, at the time, second only to London’s Great Fire of 1666. The blaze started in a warehouse in Hanover Square and quickly spread to Wall Street, burning most of the city’s warehouses, shipping offices, banks, and financial buildings. It did not reach the new Trinity Church, which had been erected in 1790, though that church would soon be pulled down anyway as part of a general post-fire building boom. It also didn’t have a chance to burn Federal Hall, because the Treasury Department—which had acquired the building in 1812 from the city of New York—had already torn it down; the vacant lot there helped act as a firebreak.

The blaze did incinerate the Merchant’s Exchange, including, despite heroic attempts to save it, the marble bust of Hamilton. When the fire finally extinguished itself, most of the city’s financial center—and its early Dutch and English heritage—had vanished.

To keep the engines of commerce humming, the city needed to rebuild quickly. Along Stone Street near Hanover Square are excellent examples of 1836 counting houses that replaced ones that had burned down in the fire. The oldest surviving Wall Street buildings are the building we now call Federal Hall National Memorial, which was originally the U.S. Custom House; it was designed by Ithiel Town and built between 1836 and 1842. Just down the street was the new Merchant’s Exchange (today’s Cipriani) by architect Isaiah Rogers, which was completed in 1841. Built at the height of the Greek Revival movement, both were modeled on ancient temples, with soaring columns and interior domes. (The Cipriani dome is gorgeously intact despite later additions to the building; you just need to sneak into the grand ballroom to see it.)

One building that escaped the fire was the Bank of the United States, which in 1822 had built an office next to the site of the old City Hall. In 1848, it became the Assay Office, where gold, recently discovered in California, was weighed, melted down, and refined. Torn down in 1912, the facade of the building now stands in the American Wing of the Metropolitan Museum of Art, the only extant example of pre-Great Fire Wall Street.

Though Wall Street trading continued to expand both before and after the Civil War, little from this era remains on Wall Street. This isn’t because of a fire or other calamity—it’s simply because everything built between about 1845 to 1900 has been replaced.

In 1865, the growth of the New York Stock Exchange led the exchange to finally build its own building, which was erected on Broad Street just below Wall. As the volume of the exchange continued to soar in the Gilded Age, this building also proved inadequate, and in 1901 it was torn down so that George B. Post could design a new home. Opened in 1903 and expanded in the 1920s, it continues to be the home of the exchange today.

Above the multistory Corinthian facade, a limestone pedimental sculpture by J.Q.A. Ward shows a female figure of Integrity, her hands outstretched over figures representing the various “works of man,” commodities that were traded inside the building: agriculture, industry, invention, mining, and science. Ward was also responsible for the statue of Washington that had been erected in 1883 across the street, in front of Federal Hall, to remind New Yorkers of the city’s role in the Revolutionary War and early federal government.

This corner—which came to be known as “the corner”—also featured J.P. Morgan’s bank at 23 Wall Street. In 1873, the building had been built for the Philadelphia-based firm of Drexel & Co., which later merged with Morgan’s firm. Known colloquially as “The House of Morgan,” the bank was so influential that it was able, virtually single-handedly, to stave off financial panics and shore up the U.S. economy. In 1882, Morgan, an early proponent of incandescent lighting, had Thomas Edison wire the building—and much of Wall Street—with electricity, ushering in a wave of technology.

A new House of Morgan opened in 1913, just after J.P. Morgan’s death. It was a pointedly low-rise building in a neighborhood that was otherwise embracing the skyscraper. Where the Woolworth Tower just up Broadway, opened that same year, was proudly the tallest building in the world, Morgan’s staid bank was essentially the shortest building in the world. For years, land on Wall Street had fetched the highest price in the city. By erecting a squat building with no rental income, Morgan was announcing to the world that he didn’t need additional sources of funding. Morgan’s bank also had no sign or outward indication as to its purpose, and lacked any of the terra-cotta design work that was the hallmark of Woolworth Tower’s more flamboyant design.

Morgan’s power led the bank to be a target. As tensions in New York grew during the era around World War I, the city was on high alert for German saboteurs, anarchists, Bolsheviks, and labor unrest. Though Morgan had been dead for six years, a gunman burst into his parish church, St. George’s on Stuyvesant Square, on April 18, 1920, and killed a Morgan family friend, thinking he was the banker.

On September 16, 1920, a bomb exploded from a wagon parked across the street from the House of Morgan. Timed to go off at noon, when many workers were on their lunch break, the shrapnel from the bomb killed 38 people and injured nearly 150 more. Though anarchists claimed credit for the explosion in an anonymous note, no one was ever prosecuted for the crime. Today, the side of the House of Morgan continues to bear the pockmarks from the blast, but these are the only visible reminder of what was, until the 1995 bombing in Oklahoma City, the deadliest terror attack on American soil.

The bombing did little to deter Wall Street’s growth. Skyscrapers had proliferated downtown since the 11-story Tower Building had gone up at 50 Broadway in 1888. On Wall Street, the Banker’s Trust Company Building at No. 16 (1912; Trowbridge & Livingston), the Trust Company of America at No. 37 (1907; Francis Kimball), and the Munson Steamship Company Building at No. 67 (1906; Kenneth M. Murchison) were soon joined by a new crop of buildings during the post-World War I building boom.

The most famous of these is probably 40 Wall Street, built as the headquarters for the Manhattan Company (the successor to Aaron Burr’s original bank), which later merged with Chase. Designed by H. Craig Severance, this was the building that was racing against the Chrysler Building in 1929 to be the tallest in the world. Forty Wall lost the contest. The stock market crash on October 23, 1929, the same day the Chrysler Building topped out, brought most construction on Wall Street to a halt for decades.

The exception was One Wall Street, designed for the Irving Trust Company by Ralph Walker of Voorhees, Gmelin & Walker. It was built atop what was considered “the most expensive real estate in New York” at the southeast corner of Broadway and Wall. Presumably, my tenth great-grandfather Nevius, who gave up this parcel of land in the 1650s, is still rolling in his grave somewhere.

When the market crashed, construction had barely begun, but the skyscraper still went up quickly, being finished in March 1931. Walker, speculating that each year as many as 200,000 people would see a building on such a prominent corner, tried to design a structure that would give viewers “mental relief and pleasure.” The most notable design element was the Red Room, mosaicist Hildreth Meière’s opulent entrance lobby.

As the Depression deepened, very little happened architecturally on Wall Street—with the exception of the replacement of J.Q.A. Ward’s pediment on the stock exchange, which was crumbling under its own weight. No doubt realizing the horrible symbolism, the Stock Exchange moved in 1936 to replace it with an enduring cast-metal copy.

After World War II, the nondescript Atlantic Mutual Insurance Company went up in 1959 at 45 Wall, and two glass-sheathed Emory Roth buildings at 100 and 110 Wall followed in the 1960s. (The latter, badly damaged by Hurricane Sandy, was converted in 2016 into the WeLive apartment complex.)

The most daring postwar building on the street is the 56-story Kevin Roche, John Dinkeloo & Associates tower at 60 Wall, originally built for J.P. Morgan and now occupied by Deutsche Bank. The classically inspired building mimics the forms of the old Merchant’s Exchange across the street, which was enlarged by McKim, Mead & White in 1907 for the National City Bank. Sixty Wall gets its excessive height from having purchased 55 Wall’s air rights. To satisfy the city’s public accommodation zoning requirements, the building features a soaring interior atrium complete with palm trees and chess tables. Until 2001, this was the best public space on Wall Street.

After the World Trade Center attack, all streets surrounding the stock exchange were barricaded, and over the last 17 years what were originally haphazard road closures have been replaced with equally haphazard, semi-permanent public amenities: decorative bollards, tables and chairs, a historic timeline along Broad Street noting its former use as a canal, and post holes on Wall Street marking (sort of) the path of the original defensive palisade.

With the streets cordoned off to traffic and more tourists visiting New York than ever—62.8 million in 2017—Wall Street is now awash in pedestrians.

The Downtown Alliance is wading into this fray with plans to remake the street as an even more pedestrian-friendly zone. This seems like it could become a case study in induced demand; indeed, the point of pedestrian-centered architecture is not merely to accommodate existing walkers, but to encourage more people to take to the streets. For example, in the 2008 pilot program for pedestrianizing Times Square, “the number of pedestrians walking through the area increased by an impressive 11 percent,” which was one of the factors—in addition to safety—that caused the Bloomberg administration to push forward with its aggressive implementation of an ever-more car-free Times Square.

It seems likely that a more pedestrianized Wall Street will be similar, though it’s worth noting that in the renderings accompanying the Downtown Alliance proposal, there are never as many people in the images as one can find on any given day snapping photos of George Washington at Federal Hall, so perhaps planners don’t want to overtly advertise this facet of the proposal.

There is already a move afoot to move the recently installed “Fearless Girl” statue from Bowling Green to face the New York Stock Exchange. The Downtown Alliance proposal clearly shows Arturo Di Modica’s “Charging Bull” also back on Wall Street, where it was originally dropped off in secret in December 1987. Considering that the bull is among the most visited and photographed sites in New York, moving it to Wall Street will surely draw even more visitors.

Since a few years before the 9/11 attacks, residential conversions downtown have been the norm, and since the city can’t seem to keep up with providing housing to match employment, it’s likely that even more buildings in the Financial District will become rentals or condos.

The New York Stock Exchange has long been tied to its rituals and traditions; the market was born on Wall Street and has stayed remarkably close to the site of its founding for over two centuries. But as the nature of trading changes, will the exchange need to maintain such an antiquated physical presence on Wall Street—or even in the city at all? It’s certainly contemplated moving before, including considering becoming the original anchor tenant of the World Trade Center. What if it moves now? If Wall Street loses the exchange, the banks and brokerage houses will follow, freeing up more residential space.

Planned well, the street could become an exciting destination, with mixed-use ground-floor retail and residences above. Planned poorly, the street might become just another faceless series of corporate chains with tourists jamming the street by day and residents barricading themselves inside at night.

Wall Street has survived attacks from anarchists and hostile foreign powers, has weathered financial panics and robber barons trying to corner the market. It is not just a shorthand for finance; it’s a crucible of New York history.

What will be the street’s next chapter?

James Nevius is an urban historian and author of a number of books about New York, including Inside the Apple: A Streetwise History of New York City and Footprints in New York: Tracing the Lives of Four Centuries of New Yorkers. He is currently researching a book about American utopianism. Follow him on Twitter or Facebook.