The maps we use to navigate have come a long way in a short time. Since the '90s we've gone from glove boxes stuffed with paper maps to floorboards littered with Mapquest printouts to mindlessly obeying Siri or her nameless Google counterpart.

The maps behind those voices are packed with far more data than most people realize. On a recent visit to Mountain View, I got a peek at how the Google Maps team assembles their maps and refines them with a combination of algorithms and meticulous manual labor—an effort they call Ground Truth. The project launched in 2008, but it was mostly kept under wraps until just a couple years ago. It continues to grow, now covering 51 countries, and algorithms are playing a bigger role in extracting information from satellite, aerial, and Street View imagery.

A few of the features that can be extracted algorithmically from Google Street View data. Google Maps

Street View, which launched in 2007, was conceived as a way to improve the user experience by letting people see what the area around their destination looked like, says Brian McClendon, Google Maps VP. "But we soon realized that one of the best ways to make maps is to have a photographic record of the streets of the world and refer back to those whenever there’s a correction," McClendon said.

And as the data collected by Street View grew, the team saw that it was good for more than just spot-checking their data, says Manik Gupta, group product manager for Google Maps. Street View cars have now driven more than 7 million miles, including 99 percent of the public roads in the U.S. "It’s actually allowing us to algorithmically build up new data layers from information we’ve extracted," Gupta said.

Those algorithms borrow methods from computer vision and machine learning to extract features like street numbers painted on curbs, the names of businesses and other points of interest, speed limits and other traffic signs. "Stop signs are trivial, they're made to stick out," McClendon said. Turn restrictions—which directions you can turn at a given intersection—are a big deal for navigation, but they're trickier to capture with algorithms. Sometimes the arrows that tell you which turns are legal are painted on the road, sometimes they're overhead. They can be different colors and sizes. "Lane markers are harder because they're not consistent, but we're getting much smarter about that," McClendon said.

Invisible to ordinary users, information about turn restrictions are built into Google maps. Google Maps

Street signs are a big deal too. Drivers can follow the app's verbal directions more easily if what they hear matches what they see. but sometimes the spelling or abbreviation used on street signs varies. "Matching what's written on the signs is actually a hard and important problem," McClendon said.

Other algorithms extract building footprints and heights from satellite and aerial imagery. The majority of buildings in the U.S. are now on Google Maps. For landmarks like Seattle's Space Needle, computer vision techniques extract detailed 3D models (see below). Google has said that its recent acquisition of Skybox, the high-resolution satellite imagery company, at least initially, is to improve the accuracy of its maps.

Google uses computer vision techniques to extract 3D models of landmark buildings from satellite and aerial imagery. Google Maps

Yet satellites and algorithms only get you so far. Google employs a small army of human operators (they won't say exactly how many) to manually check and correct the maps using an in-house program called Atlas. Few people outside the company have seen it in use, but one of the most prolific operators on the map team, Nick Volmar, demonstrated the program during my visit. (There's also a fascinating demo in this video from Google's 2013 developers conference).

What the operator sees looks similar to the hybrid satellite-map view in Google Maps, but with unfamiliar colored lines and symbols. Roads, for instance, are color-coded according to the direction of travel. Green and red arrows indicate which turns are possible from a given intersection. Volmar deftly clicked boxes on one side of the screen to toggle various layers on and off. Traffic signs captured from Street View imagery appeared and disappeared.

Volmar showed how an operator could fix a road that's out of alignment with the satellite image by clicking and dragging it into place. It looked easy, maybe even fun, and not unlike the process for editing Open Street Map. Volmar and other operators also check out tens of thousands of problems reported daily by Google Maps users and fix them as needed.

Operators can highlight a road (top left) and drag it into alignment with satellite imagery. Google Maps

One map showed road prioritization for a major throughway in San Francisco, with the width of the lines representing the volume of traffic. Google has been using cellphone location signals to map traffic conditions for years, and Gupta acknowledges that location signals could also be a good source of other information, about turn restrictions, say, or one way streets. But he declined to elaborate. "Google uses location in multiple ways, but there's nothing specific I can talk about beyond that," he said.

In addition to operators like Volmar, Google also gets cartographic help from ordinary citizens via its MapMaker program, which launched in 2011 and now operates in more than 220 countries. The goal was to improve Google's maps for developing countries and other areas where accurate and detailed source maps weren't available. "We recruited users to add mapping info where it's important to them," Gupta said. "We provided a tool and good satellite imagery so people could trace on top of it."

Closer to home, people can contribute data on parks, trails, and other places Street View cars can't go. McClendon has mapped the hiking trails at Windy Hill, a popular nearby hiking spot. "I GPS'd my way up that hill and added more precise trails," he said.

Watching the screen as Volmar breezed through the Atlas demo, it was easy to get a sense of what The Atlantic called the "deep map" underlying what you actually see when you pull up Google Maps on your laptop or phone. There's far more data beneath the surface, information not just about the layout of roadways but also the logic of how you'd connect from one to another. Information not just about the shapes of buildings, but, increasingly, what's inside. The maps, it seems, are only getting deeper.