I hadn't signed up for this. A small corner cafe in the Bolivian city of El Alto and a steamy bowl of soup plonked down in front of me. The dish in question, a local speciality known as caldo de cardán, sounded fine until I heard the English translation – bull's penis soup.

I think the owner of the restaurant had taken pity on the perspiring gringo seated in his establishment and included only a modest phallus in my portion. Now all I had to do was give it a try. Spoon at the ready, I chickened out on first attempt after the diner at the next table explained – with a certain relish – that the particular tint to the soup was thanks to the bullish offering. Second attempt and I managed to get it into my mouth. The taste wasn't unpleasant but the gristly texture did for me.

Within Bolivia, caldo de cardán is famed for its energy-boosting qualities, and for doing wonders for male fertility. But among foreigners the country is hardly feted as a gastronomic mecca. Visitors tend to come for the scenery. As for the grub? Forget it. In recent years, that job in Latin America has gone to neighbouring Peru, with is delicate flavours and high profile celebrity chefs such as Gastón Acurio, who is due to open a London restaurant later this year.

But if Bolivia is meant to be a land of bad food, Claus Meyer – founder of the Copenhagen's Noma, voted the world's best restaurant for three consecutive years until it lost the crown this year – clearly hasn't been listening. In April, the chef-turned-business mogul opened Gustu (restaurantgustu.com, dishes £6-15), in the southern part of the capital, La Paz, to prove that Bolivia can give Peru a run for its money when it comes to cuisine. I'm here on a gastronomic tour to see if this could happen.

A good place to start is La Paz's Mercado Rodríguez on a Saturday morning. It is packed with vendors and punters, and the view is stunning in this city of almost impossible gradients. I'm standing in the city centre, 3,600m above sea level, and I feel I am almost touching the low-hanging fluffy clouds that stretch to the horizon.

Bovine delight …Caldo de cardán, or bull’s penis soup. Photograph: Ed Stocker

The market is flooded with colour. Makeshift stalls are tended by cholitas – the local term for the rural Aymara women in traditional dresses and bowler hats who have come to sell their produce. I have managed to steal Mauricio López, one of the Gustu sous chefs, to explain the weird and wonderful produce on display: fruits and vegetables I simply didn't know existed, alien shapes and forms, undiscovered tastes. Pacay, for example, is a green-husked fruit with giant black seeds and a texture a bit like candyfloss inside the thick pod.

And then there are the endless varieties of potatoes – I'm proudly told Bolivia has more than Peru – of which the most spectacular is the papalisa, an incredible spud that looks more like a fruit than a familiar stodgy staple, its bright yellow skin covered with fluorescent pink spots.

We arrive at another potato stand – with a difference. All of the produce here is covered with what looks likes a white powder dusting. It turns out that this is the tunta, cousin of the jet-black potato I'd tried in the, ahem, aforementioned soup.

"In the Altiplano, people place these potatoes in sacks and then put them in a river to wash through," explains Mauricio. "The skin is removed and then they're left out in sub-zero conditions overnight. There's no moisture left after this process; they can last like that for years."

It's this hotchpotch of exotic fodder that Gustu has tapped into. I head there in the evening to try some of the produce I've been staring at during the day. The modernist space is in the Calacoto neighbourhood, one of the wealthier parts of La Paz, a city where the poor look down on the rich. The service is super-slick, marshalled by the amiable maître d' and sommelier, Jonas Andersen, a Dane who towers above me.

With chefs from Venezuela and Denmark respectively, there's a definite international slant to this restaurant – and it wouldn't look out of place in London or New York – but all of its ingredients are sourced from within the country.

The food is beautifully presented: minute chicken hearts rolled in a spicy peanut cream; shredded Huaycha potato, poached trout from Lake Titicaca with coa, a unique minty herb; and quinoa (Bolivia is the largest producer in the world of the superfood du jour) with rehydrated cherries, amaranth (another local plant) and buttery walnuts. The 15-course tasting menu, with an alcohol pairing for each course, will set you back 925 bolivianos (around £87; the five-course version costs £39). Though far less expensive than the equivalent fine dining experience in Europe and the US, it is still way out of reach for most of the Bolivian population.

Claus Meyer (left), founder of Noma, in the kitchen at Gustu. Photograph: David Mercado/Reuters

But Meyer's project isn't intended simply as a moneymaker. Staff at the restaurant often come from humble backgrounds and are there to learn everything from food hygiene to how to how to fillet a fish. Meyer has set up the Melting Pot Foundation, a philanthropic arm aimed at teaching food education. The foundation takes a cut of profits and dividends from the restaurant to plough back into the local community.

"Even though I'm Bolivian, I've tried ingredients here that I hadn't even heard of," Estefanía Morales, a 22-year-old trainee, tells me before scurrying into the kitchen. "My mother sells food in El Alto and now I'm telling her 'no, you have to do it like this'."

The food boom in La Paz is by no means restricted to Gustu, however. Steady economic growth and an expanding middle class mean more expendable income kicking about, and new restaurants are opening all the time. My tour of the city also includes a visit to Villaserena (Avenida Ecuador 2582, +591 2 241 8151, mains £5-10), a bohemian joint in an old building that also functions as a cultural centre. The food here is very much high-end gastronomy, but the setting is wonderfully laid-back with chef-owner Juan Pablo Villalobos wandering around his dinner guests' tables. I tuck into an intriguing salad of ajipa, blueberries, raisins and herbs. Ajipa is another otherworldly vegetable, the shape of a sweet potato; peeling it reveals a white vegetable with purple flecks. Its taste isn't strong, and it has the texture of an apple, perfect with olive oil and sweet fruit.

Another highlight is his particular take on sajita de pollo, a classic Bolivian dish of spicy chicken that normally comes with a sarza, a garnish of red onion and tomatoes. Here, though, Juan Pablo has swapped the tomatoes for strawberries.

As I am finishing off my dinner, the distinctive chords of Bon Jovi waft towards me. Juan Pablo also owns the next-door Full Moon rock bar, and plays bass guitar. He is gagging to take off his whites and change into black, let his locks down and join his fellow rockers on stage. Before I leave, he presses a CD into my hand, the latest from his band, Tapako. The album is daintily entitled Drunk Punk Death Doom Fuckin' Metal.

Ham production in Bolivia’s Tarija region is a legacy of Franciscan missionaries. Photograph: Ed Stocker

Another rather good establishment in La Paz is El Vagón del Sur (Avenida Julio C Patino 1295, +591 2 279 3700, mains £6-£9.50) in the south of town, which again focuses on Bolivian dishes with a fusion twist. But perhaps my favourite place is run by a Swiss German, Walter Schmid, who married a local woman and settled in this country more than two decades earlier. His Oberland restaurant and hotel (Calle El Agrario 3118, Mallasa, +591 2 274 5040, h-oberland.com, mains £4.50-£6.50), is a calming haven in what can be a mad city, with tables and chairs in a tranquil garden space. Oberland is in the suburbs of the city, at the edge of the Moon Valley where cliff erosion has left epic, lunar scenery. His establishment may be built like a Swiss chalet but he knows a thing or two about Bolivian food: his quinoa tabbouleh and tumbo sour – an aperitif of tumbo fruit and singani, the country's potent grape-based spirit, similar to Peruvian pisco – are particular winners.

But as surprising as all the delicious grub is the way the drinks match it step for step – not just the singani but the wine, too. Prior to my visit I wasn't even aware that Bolivia was a wine producer, but I find myself taking an internal flight to its vine-growing heartland, the southern city of Tarija. There have been vineyards here since the 17th century but it wasn't until the 1960s that the industry took off, with the introduction of modern wine-producing techniques.

The town, just a few hours from the border with Argentina, is a major domestic gastronomy destination thanks to its vineyards and, surprisingly, a booming cured ham market – a legacy of Franciscan missionaries in the 18th century.

I'm on the Ruta del Vino y Singani de Altura, a panoramic route between bodegas in one of the highest-altitude grape growing regions in the world, stretching up to 3,330m. I've prepared myself for some rather sweet, poor-quality wines – and am about to be proved completely wrong.

We drive out of town, past the Guadalquivir river, and into a desert-like terrain of yellowy-brown soil and thick shrubs. Clouds hang over the nearby mountains. My destination is Bodega La Concepción (+591 4 663 2250, laconcepcion.bo), 2km from the village of La Concepción in a valley of the same name, the oldest winery in the region and located on land formerly owned by Luis de Fuentes, Tarija's founding father.

Guests are received on a beautiful terrace, with a table set up next to the vines for tasting. Of the several surprisingly good varieties I taste, the best is the Cepas de Altura line and the 2009 syrah, oak-aged for three months with a slight kick – but not as heavy as some other South American reds. The whites are pretty decent too.

With a booze buzz on, it's time to move on to Bodega Campos de Solana (+591 466 48482, camposdesolana.com), in the Santa Ana valley closer to Tarija, and probably Bolivia's biggest wine player.

"Wines from around here are similar to those from some zones of South Africa, or Patagonia. There's a decent fruity presence and the taste isn't too concentrated, which means they're very drinkable," oenologist Nelson Sfarcich tells me as we quaff the 2010 Trivarietal Reserva, a blend of cabernet sauvignon, malbec (Argentina's grape of choice) and tannat, which is popular in Uruguay.

Woman harvesting cabernet sauvignon grapes in a vineyard in Tarija, Bolivia. Photograph: Peter Langer/Design Pics/Corbis

Tarija is also the birthplace of singani, and Nelson offers me a shot of the heady spirit before we move on to the longest-established of the white spirit's producers, Bodegas Kuhlmann (+591 4 664 4346, bodegaskuhlmann.com), which has been making the Los Parrales brand of singani since 1930. Co-owner Franz Molina, who looks like a bodybuilder, gives me a tour and shows me the distilling process. Singani has a clean, fragrant kick and deserves its place on shelves at cocktail bars around the world. Franz tells me plans are under way for a regional denomination, in order to help promote the tipple.

The bodegas of Tarija may not yet have the infrastructure of Mendoza or Cafayate in Argentina – and the internal market may also be small in comparison– but the potential is there. Things are beginning to change, too. There's a project to create a wine-related art installation on one of the bridges along the wine route, and La Concepción will open a guesthouse later in the year, so tourists can stay out in the valley instead of returning to the city with their tour or hire car.

As the day draws to a close it is time to bring my Bolivian degustación to a close and return to Tarija. I head into town to try out the street food. I've had a fair amount of booze and there's nothing like saice, a Tarijan speciality, to soak it up. The juicy minced beef and peas with rice is just as worth eating as the high-end cuisine I had in La Paz. I also sample skewers of tripa (yes, tripe) from the next stand. I'll happily try anything, I think to myself, but then remember the one thing I drew the line at: caldo de cardán. The less said about that the better.

Bolivian women harvest potatoes in El Alto. Photograph: Ali Burafi/AFP/Getty Images

Standing at 3,600m above sea level at its centre, La Paz is one of the highest cities in the world, and you'll need to take time to acclimatise on arrival. One way to do this, locals will tell you, is to drink mate de coca (coca tea), made from leaves of the coca plant and consumed across the Andes, especially in Bolivia, Peru and Argentina. The locals, of course, are already well acclimatised. Bolivia may not have a very successful national football team but when visiting teams come to play in La Paz – even football giants Brazil and Argentina – they often lose because of the difficult conditions.

Cooking at altitude also has its complications. Water boils at 88C, so while a cup of tea tastes fine, rice takes a lot longer to cook; salt reacts differently, so you need to season dishes more; and bread is reluctant to rise. When it comes to alcohol, pouring a beer produces more foam because there is less air pressure. And then there are the stonking hangovers …

The bitterly cold nights on the Altiplano are also perfect for giving potatoes that are left outside a rich, musty taste. The chuños and tuntas (see main text) – frozen potatoes that are later rehydrated – wouldn't be the same without the particular cold, dry climate. One favourite local dish involves boiling the potato with anise the next day and serving it with egg.

There are other foodstuffs that just wouldn't be the same without the altitude. In the manufacture of pasankallas, the puffed white corn that is Bolivia's super-sweet take on popcorn, the low air pressure allows for perfect popping. One pasankallas company moved its factory to El Alto, a satellite of La Paz, for that reason.

Standing at an incredible 4,150m above sea level, El Alto is the world's highest large city. But altitude isn't the only unique thing about El Alto: with a population of almost 1.5 million it is the largest Amerindian city in Latin America – about 76% of its inhabitants are Aymara, and less than 1% are of European descent.

• The trip was provided by High Lives (020-8144 2629, highlives.co.uk). Its seven-day Gourmet Tour of Bolivia costs from £3,000pp, including domestic and international flights from Gatwick; visits to wine producers, salt producers and quinoa plantations; a desert safari; and the 15-course dining experience at Gustu. Flights from Gatwick to Santa Cruz via Madrid were provided by Air Europa (0871 423 0717, aireuropa.com, from around £750 return). Internal flights were provided by the local airline Amaszonas (amaszonas.com)