Government Island, Virginia (CNN) About an hour's drive from Washington, a quiet, suburban neighborhood gives way to an unassuming gravel parking lot. Neighbors meander on a newly constructed boardwalk with their dogs and strollers, passing over a creek until the deck gives way to Government Island. Today, it's a peaceful Stafford County park with a mile and a half of wooded trail, rocks, plants and wildlife. But in the 1790s, it was a hub of activity: This island provided the stone for the White House.

Historian and author William Seale has been coming to Government Island since 1974, when he began writing a two-volume book, "The President's House," about the White House. Back then, before the boardwalk was built, he would trek through mud and the knee-deep creek to get to the island, which has otherwise been untouched since its heyday.

"This is like going into a ruin, a beautiful ruin where people have just abandoned it for other lives. They don't need it anymore. The kind of things that evoke thoughts of the past, and it's romantic and, in its little way, it's authentic," Seale told CNN during an October walk through the park. Seale wrote a book this year about the process, "A White House of Stone," for the nonprofit White House Historical Association.

"All that people had were hands and oxen and a boat," he said.

City built of stone

As George Washington envisioned his plans for a capital city in the late 18th century alongside designer Pierre Charles L'Enfant, he wanted a city built of stone. At Washington's direction, L'Enfant purchased Government Island, then a small quarry on Aquia Creek, which runs into the Potomac River. More than 100 men cleared the island of its trees and brush and began what became a major production of Aquia sandstone lasting nearly a decade, from the fall of 1791 until the end of 1799.

The rock that wasn't harvested, and some of the smaller split stones that were rejected for their imperfections, still stands at Government Island -- complete with visible chisel marks from where it was split more than two centuries ago.

The workers, mostly slaves, were employed through contracts from their masters. Freed men and stonemasons from the US and Scotland were also on site. They constructed small cottages to live in, and one big fully staffed kitchen house.

"It was a whole process of hand work that took the rock from Government Island here to Washington to build the city that is now the wonder of the world," Seale said.

Extracting stone from walls of rock was an arduous task.

"How they did it was they used a green pole about 5 feet long, a green stick stripped of any leaves. They drove it down into the stone when they would find a place they could do that, then they would pour water on it continually, then the stick would expand and split the stone," Seale said, explaining this could take weeks.

Then oxen would haul the heavy stones to the edge of the island. From there they were loaded onto a shallow barge and paddled upstream to the Potomac, where they were loaded onto another boat. Once the stone had made the 40-mile voyage to Washington, it was unloaded through a temporary system of locks to the building site, near where the US Treasury building now stands.

'There was no guesswork'

After arriving at the building site, the stone was meticulously refined and numbered before it was taken to the White House and hoisted onto the walls with a simple crane.

"It was all laid out on the ground at the White House where that part of the wall would be, and the stones were numbered, and that's how they would be set. There was no guesswork; they knew where the stones went," Seale said.

Once in place, the stone was whitewashed to prevent cracking. (When the White House burned in 1814, most of the original stone survived but had to be painted with white lead paint to cover up the soot.)

"It's real nostalgic, you know, to think of what went on here, and hundreds of people working and the building they built that now is an icon in the world. You think about all those things. It's where it began. And it could have failed, what it built could have failed, and it didn't," Seale said, sitting on one of the rejected stones.

Washington died shortly before the building was completed.

"Washington envisioned a great world power and he wanted it to look like a world power, and these (stones) are the tools of one aspect of his attempt to realize that," Seale said.

The first US president initially wanted the White House to be five times the size it was ultimately built.

"His advisers said, 'This is impossible; we'll never get enough stone.' So he agreed to reducing, but he said it could be added on to in ages to come. So he was a visionary, and he got the nut of the thing planted," Seale said. "I think he was at peace with his city. I think he got what he wanted."