Google is building the Internet’s most comprehensive map, one backpack at a time. Tom Vanderbilt joins the tech giant in St. Croix to see how it’s recording the world’s remotest corners—and in the process redefining how we get from one place to another.

In the gravel parking lot at Rainbow Beach, a popular, calm-watered spot on St. Croix, in the U.S. Virgin Islands, Google’s Mara Harris is struggling to stay upright. Her horse, Firefly, keeps nervously shifting and stomping. “She’s probably not used to the extra weight,” says Jennifer Olah, proprietor of Cruzan Cowgirls, a local horse-rescue outfit that runs beach riding tours. The extra weight is a 40-pound contraption strapped to Harris’s back: the Google Trekker. The custom camera rig, which captures imagery for Google’s Street View program, sits in a military-style backpack and has a towering extension topped by a large green orb. People often mistake it for a jet pack.

I take all this in from atop my own horse, Crow, who is more concerned with munching grass on the sly than with Harris’s curious appendage. But then, the clouds that had suddenly gathered begin to yield drops. A Google technician scrambles to retrieve a black garbage bag to protect the Trekker’s lenses. As it turns out, if there’s one thing horses like less than an extra-heavy rider wearing a robot pack, it’s having a plastic bag flapping above their heads. Firefly bolts, circling in the parking lot, Harris and the Trekker nearly tipping off before she pulls off an admirable save and rights herself.

We are on St. Croix because the largest of the Virgin Islands is nearly impossible to navigate using Google Maps. “Right now, if you tried to get to the airport, it would put you into Estate Kingshill—which is not where the airport is,” says Kirk G. Thompson, the local point man on Google’s effort to map the island, over coffee at the Avocado Pitt in Christiansted, the island’s biggest town. “And an airport’s a rather large thing to miss!”How about the botanical gardens, one of the island’s Top 10 destinations? “That sends you somewhere in the rain forest.” Thompson worked for tech and outdoor companies (with a stint at the U.S. State Department) on the “mainland” before coming to St. Thomas, then to St. Croix, where, he jokes, “St. Thomians come to relax.” He co-owns N2 the Blue, a popular dive shop in Frederiksted, which is how he came to be so deeply involved in Google’s mapping effort. As he points out, instead of giving exact addresses, Cruzans tend to say things like, “take a right at the pink house,” which works fine—until the house is painted blue. But if we were to open Google Maps and try to use it to find our way across the island, we would have very little chance of success. He gestures to a colorful tourist map in a rack of brochures. “That’s more accurate.”

It’s hard to overstate the ubiquity of Google Maps in the modern world—and never more so than when we travel. One in five Google searches is related to location, and the figure jumps to one in three when the search is performed on a mobile device. When you take an Uber to your Airbnb (which you cased on Street View) and then find a place to eat using TripAdvisor, you’re relying on data from Google Maps. On a recent hike to Finger Rock, in Arizona, I lost the trail. I stumbled about for a while and was about to turn back when, in a moment of casual desperation, I opened Google Maps. I (or the little blue dot that was me) was just a bit to the left of the trail—faint in real life but ridiculously clear on the map.

Anyone from pro photographers to travelers can apply for a camera loan.

The map is not the territory, goes the old saying, but the app is the world. And yet Google Maps is far from infallible. It seems like every week there is a story of tourists who were directed into a private driveway instead of a national park. In 2010, Nicaragua blamed Google Maps for a misguided invasion of Costa Rica. For someone like Thompson, an unreliable map is just bad for business. “You might have had the greatest time diving, or horseback riding,” he says. “But if you get lost, it might sour the entire experience.” In other words, as our faith in Google Maps grows, when we do get lost, we might subconsciously take it out on the destination. Travelers now expect that punching “coffee” into Google Maps will display all nearby cafés. The stakes for business owners are huge. Not being discoverable on Google Maps is like not being discoverable in a Google search: you are, in essence, invisible.

Increasingly, our movement through time and space is mediated via screens. As happy as I was to be put on the right path in Arizona, the episode raised the very real question of what dependence on smartphones does to us. Research has shown that people who relied on GPS were less able to draw the cities they lived in to proper scale, and less able to create a mental map approximating where they had been. Maybe I have already lost some of my ability to detect objects in the real world, like trail markings.

We travel to see new places, but this all-powerful compass may confine and restrict our knowledge. One of the ways we learn, after all, is by making mistakes. Being somewhere new should mean stumbling into places by accident, or dealing with locals in an unfamiliar language, or even being so immersed in a place that it doesn’t really matter if you lose your way. Yet even when we want to travel off the beaten path, we still expect to be shown exactly where that is. And maybe that frees us from the anxiety and hassle of finding our way, allowing us to be more in the moment.

In the early days of Street View, the emphasis was on “giving you a virtual travel experience,” says Luc Vincent, a senior director of engineering for Google and the lead engineer of Street View. The program has evolved from those somewhat gimmicky origins; the imagery acquired from Street View now actually makes maps better. The ante was upped further as users began to expect not only accurate maps of their destinations but panoramic images as well. “People wanted to see more detail—not just the street, but the pedestrian walkway and every store in a mall,” Vincent says.

As Vincent tells me, Street View grew from company cofounder Larry Page’s desire to “not just bring the Web to our users, but bring the world to our users.” After the technology was condensed into backpack size—opening up all those places inaccessible to a Street View car, like the face of Yosemite’s El Capitan, the Burj Khalifa skyscraper, or the “secret cave” of MythBusters cohost Adam Savage—the company realized “there was no way we could do it ourselves.” They began lending the Trekker to governments and tourism boards. “We wanted to engage communities and partners who know places better than we do and who are passionate and can bring those places to life through maps.”

Thompson—who is nothing if not passionate—is the first individual Google tapped to map a territory, in part because he had already been making so many additions as a civilian to Google’s St. Croix maps and Street View. Any user can add to maps, though that has led to the addition of some controversial, and since removed, material. Google says that anyone—from pro photographers to ordinary travelers—can apply for a camera loan. Illustration by MCKIBILLO

Thompson is essentially responsible for accurately mapping St. Croix— and by extension, creating the virtual experience for those who take a digital sneak peek before they visit. Over the past months, he has become a familiar sight on the island, conspicuous with his hiking poles and 40-pound appendage, roaming through botanical gardens: dutifully tracing 18-hole golf courses; plodding about old sugar mills; and embarking on a five-mile hike around Point Udall, the easternmost point of the territorial United States, in full Caribbean sun and humidity. He walked from the end of the pier in Frederiksted and into town so a theoretical visitor from a cruise ship (the island gets about 50 ships a year) might preview the experience.

Street View has been deployed, at least partially, in some 77 countries—its website lists an extensive number of locations around the world the company plans to map next. But places like St. Croix show vividly how daunting a challenge Google faces: the island’s quirky house-numbering system, the road closures and other physical changes to the landscape wrought by devastating storms, Google’s reliance on existing third-party map products. Take all those factors and multiply them by the number of remote unmapped destinations in the world, each with its own idiosyncrasies, and you get an idea of the difficulty. In a moment of cosmic irony during our filming, Mara Harris tries to use Google Maps to guide a taxi driver to her Airbnb lodgings; when that fails, she has to pull up a satellite image on her iPhone.

Capturing these photos takes training, hard work, and optimal conditions (for usable photos, Street View must operate at midday, when there are no shadows present, and in clear weather). Our plan one afternoon to film the gorgeous shoreline of Buck Island is canceled due to rain. We leave the Trekker behind and go snorkeling instead, under the pier in Frederiksted. The vista rivals anything above water: an endless array of deep pillars, stretching into the pale blue murk, each encrusted with a staggering array of coral and marine life, from sea horses to frogfish (a fish that looks, er, rather like a frog). Soon, this view may too be a click away—Google worked with the Catlin Seaview Survey, which tracks coral-reef loss and publishes breathtaking “virtual dives” to document Australia’s Great Barrier Reef with a special underwater camera.

It’s hard to overstate the ubiquity of Google Maps when we travel.

I am reminded of a line in Greg Milner’s recent history of GPS, Pinpoint: “In an age when GPS gives us a blue dot on a map—and perhaps also a rich visual image to go along with it—it becomes increasingly difficult to understand that this system is imaginary.” For one, we rarely stop to question whether the data might be wrong. Maybe someone entered an incorrect data point; maybe the image was taken a while ago and now there’s a busy overpass running past that quaint rental apartment. But, more broadly, it suggests that our online tools can only ever be a simulation. The blue dot on a location-aware map is a powerful, egocentric metaphor; the 360-degree panorama of the Djemaa el-Fna, in Morocco, may help you feel as if you are there. But you will never actually experience a place until you are there. As Alain de Botton puts it in The Art of Travel, “we are inclined to forget how much there is in the world besides that which we anticipate.”

Are we also inclined to forget the joy of serendipity and discovery (and not the “discovery” coughed up by an online algorithm)? As the Google Sightseeing website—an unaffiliated site that takes users on world tours with Google’s mapping tools—cheekily asks, “why bother seeing the world for real?” When I ask Vincent if all this information will dampen our desire to travel, he thinks for a moment, then brings up the case of Pompeii. “It was one of the first places we sent the Street View tricycle,” he says. “We were told by the Italian authorities that foot traffic has increased substantially since we launched the imagery.”