There was a strict warning not to carry more than five kilos of baggage on the little twin-prop plane from San Andrés to Providencia and I complied: what do you need on a tropical island, apart from bathers and a few T-shirts? But that was the sole regulation, and the customs officer at San Andrés airport happily waved me and my large glass of rum and coke through. His British counterpart would have tazered me for carrying a bottle of water.

It was a fitting prelude. Twenty minutes later, a few steps from Providencia's short airstrip, I waved down a man in a big Chevy who took me, ever so slowly, to the little chalet I'd booked from England. He was a taxi driver, I guess, but he had no badge, no rooftop light and no urge to hustle for fares. He said his name was Fed. We made small talk and he remarked that he couldn't cope with the hectic pace of San Andrés.

To put this into some kind of context, mainland Colombia is 100 times more laid-back than the UK. Its most touristy Caribbean island, San Andrés, is a 12km-long dot some 775km to the north of the mainland, closer to the coastline of Nicaragua than to Colombia, and it's best-known as a destination for divers and well-off families. On my way back I spent a few hours there: it was convivial, full of beach bums, hassle-free.

But Providencia – which sits alongside San Andrés and is a third smaller – is something else. It's not just the absence of jets. There are no millionaire yachting types, no all-inclusive resorts, no boat parties at dusk. The lack of a proper airport keeps the island small and package-tour-free. It is also culturally separate: Creole English is still the dominant language (locals call the island Old Providence); Colombia's capital, Bogotá, has tried to impose Spanish; and the population – and the local government – is black and of African descent.

Providencia hasn't got the restaurants and resorts of Antigua or Jamaica. What it offers instead is a sort of do-it-yourself Caribbean experience. Shortly after checking in to my chalet I heard a loud thud – it was a windfall of mangoes landing on my roof. I grabbed a couple and went down a small staircase to a jetty. There was no one there – no boats in the water, no bars or kiosks. I jumped in and swam in the tepid water. Seeds from a cotton tree were floating all around, and beneath me hundreds of little black and yellow fish swam about. Other than that, nothing.

My villa was part of a small complex run by a local firm. We could have lunch or dinner there, or not, but one meal a day was included, and as you could bring your own booze it seemed a good deal. On the first afternoon I strolled up to Morgan's supermarket and picked up some decent Chilean wine and a bottle of Caldas rum. Over the days I got to know the supermarket ladies, and tried their fried meat patties. The only other shop, which I popped into during the evening, was a French-owned arts and crafts place that did good coffee, local moonshine and tasty fruit jams.

I'd gone to Providencia with no preconceptions. Indeed, I'd made the convoluted trip to this corner of the Caribbean partly because I'd never read anything about the island in the travel press. Yet, by the end of the second afternoon, as I lay back in my hammock with a rum and coke in one hand and my copy of Killing Pablo (Mark Bowden's biography of the Colombian drug lord Pablo Escobar) in the other, I was thinking that I had landed, by chance, in paradise.

On a slow (everything in Providencia is slow) cruise by launch around the island's 20km of coast, I got to see the few highlights – rustic Manzanillo, the smarter South West beach, and the "capital", Santa Isabel – as well as the stunted extinct volcanoes that run down its spine. There was only one, biggish hill. Its name: The Peak.

We also stopped by Santa Catalina, a tiny island connected to Providencia by a raised malecón or boardwalk known as Lover's Bridge. Here, the Rastafarian guide gave us some cursory history: the famous Welsh privateer Captain Henry Morgan – nephew of the lieutenant-governor of Jamaica – came to the archipelago in 1665, and in tribute to him two rocky formations were known respectively as Morgan's Head and Morgan's Buttocks.

"Morgan planned his raids on the Spanish empire here, and people still say that there's treasure hidden," he said. "But the British took it all, no doubt." He added that it was Morgan who frightened away the Armada, keeping up the link with Jamaica and Africa.

The guide also pointed out some churches – Baptist and Catholic – and a statue of Mary. "The island's only virgin," he quipped.

It is a scruffy sort of island – at least on an overcast day – and the dusty, rain-starved slopes are in harmony with the shabby British colonial-style homesteads. You still see pockets of traditional architecture in other Caribbean islands, but Providencia has had no encroachment from chain hotels or large-scale developments and, no doubt helped by its Unesco designation as the Seaflower Biosphere Reserve, it still looks raw and inviting.

At the north-eastern tip of the island, however, is a lump of rock that ticks any brochure boxes anyone might have in mind to fulfil their Caribbean fantasy. Just a hundred metres or so in circumference and as tall as a three-storey house, Cayo Cangrejo (Crab Cay) is a pristine perfect islet covered in virgin palms and surrounded by rings of coral beneath shallow seas. To work up a little sweat, I ran up to the top of the islet and was rewarded with a 360-degree view of an expanse of sea, changing in hue from turquoise to emerald to royal blue, depending on the water's depth and the shadows cast by the cumulus clouds above. I'd seen it from the plane, of course, but now it was right there beneath me, demanding exploration.

I grabbed a snorkel. As part of the Parque Nacional McBean Lagoon, the waters here are filled with protected coral reefs and mangrove swamps (the area has the world's third-largest barrier reef), as well as lobsters, sea snails, bream, grouper, crab and dozens of other reef species. I swam slowly through bath-still water and, even when I arrived on the windward side of the island, the current was barely a gentle push against my swimming stroke. There were sponges, anemones, cardinal fish, parrotfish, angelfish, squirrel fish, and huge starfish. A large, beautiful sea turtle scooted off when he saw me round one corner.

You could use scuba gear here but frankly, why bother? Visibility of 30 metres is no exaggeration and the water is about three metres deep – so I dived without the snorkel to see the starfish close up. There was no need for mechanical clutter or compressed air.

The rest of the cruise around the island couldn't match Cayo Cangrejo for wonders, but it was never less than deeply pleasant. We stopped at beaches for swims and cocktails and fresh seafood. Reggae was playing in most of the beach shacks – Colombia loves its salsa and cumbia music, but here the locals still think of themselves as fully Caribbean. The sand was gritty and rough, so on one beach we did some DIY exfoliating; elsewhere in the Caribbean you'd go to a spa and pay for some variation on this. Then a storm rolled in and we swam in the surf under the rain, with a magical yellow-grey sky above us and a strange calmness in the water even as powerful gusts skimmed over its surface.

That evening, back at the resort, armed with my bottle of Chilean Sauvignon Blanc, I strolled down to the open-air restaurant for a dinner of grilled grouper with rice, beans and fresh salad. Some kind of variation on this was, to be honest, the only dinner the place offered, though one evening I requested a spicy sauce and the kitchen prepared me a sort of jerk dip. It wasn't fiery hot like the stuff I've eaten in Jamaica and Brixton; the people of Old Providence don't much go in for being startled by food, or by anything else.

Despite greyish skies, I'd acquired a serious tan, or at least the glowing first phase of one. May is low season, when Colombian kids are at school and the seasonal rains begin. But the few showers I experienced were very welcome, falling as they did in the late afternoons when the cicadas began to panic.

Providencia is, generally, off the hurricane trail that regularly sweeps through Cuba, Florida and the Antilles, and the low-slung buildings and vernacular architecture withstand storms better than the Cancún-style tower blocks that dot the rest of the Caribbean.

Over four days, island life lulled me into a pleasant routine. Everyone travelled around on little scooters and dinners were plain grilled fish with rice and beans. Locals on the nearby beach had a smile and time to chat, between caulking boats and mending fishing nets.

There was, however, one minor miracle to observe come nightfall. By a fluke I'd arrived during the crab migration, and every night at around 7pm, hundreds of thousands of black crabs – which are huge, and frighteningly clawsome – made the trip from the woods to the beach to lay their eggs. An army of them hobbled across the main coast road – a barrier kept the cars away – occasionally scrapping for a patch of land or struggling to climb a kerb. The spectacle was endlessly engaging.

Providencia was living up to my initial impression of paradise. But, of course, all Caribbean islands try to sell us the fantasy of an Eden on earth. Local tour guide Jennifer Archbold Ramirez explained to me, "People are always trying to buy the land here, so they can build multi-story hotels and flash resorts, but we are fighting to keep them out. It's not easy, but we want to manage this at our own pace, in our own way."

Providencia was all about pace. I really don't remember the last time I switched off to such a degree. Imagine a place with no mobile reception, no internet cafe other than in the main urban area, no shops, no chains of any kind, no chefs, no TVs in the bedrooms. Less is more, nothing is everything. Providencia by name and by nature. Colombia being off the map for so many years has ensured this island's isolation from Latin America's anxious, often half-baked urge for European-style development.

On my final morning the windfall mangos woke me up, and Fed came to drive me to the airport. To my small bag of T-shirts and bathers I had added a pot of tropical fruit jam. In Madrid airport on the way back, a customs officer took my jam off me. She was a jobsworth, a European, the sort of Spanish bureaucrat that built Latin America. They never got Old Providence, though, and now I toast that gorgeous little island, as well as Henry Morgan's head and buttocks, every time I pour a rum and coke.

Essentials

Journey Latin America (020 8747 8315; journeylatinamerica.co.uk) offers a seven-night holiday visiting Colombia's capital. Bogotá, and Providencia from £1,657 including flights from the UK, half-board accommodation in Providencia, B&B accommodation in Bogotá and airport transfers.The crab migration takes place in early May and lasts around two weeks.

More treasures of the Caribbean

Isla de la Juventud, Cuba

Cuba's second-largest island lies directly south of Havana. The "Isle of Youth" is famously friendly and it's easy to get off the beaten track. Punta Francés is a top scuba-diving spot and there are swathes of wilderness in the southern half of the island. Hotels on Isla de la Juventud are limited so it's best to use one of the homestays (casas particulares) in the capital, Nueva Gerona.

• Local airline Cubana flies from Havana for $80 return. A two-hour catamaran trip from Batabanó, 44 miles south of Havana, costs $50 return (00 537 834 4446; cubana.cu); casaparticular.com links to hundreds of accommodation websites.

Puerto Rico

Since the embargo on tourism in Cuba, Puerto Rico has been middle-class America's main Spanish Caribbean option. The capital, San Juan, is a vibrant city with a pretty colonial core and while it has plenty of shopping malls, luxury hotels and glitzy casinos, it also has cool restaurants and great nightlife. In the interior, there are crumbling colonial towns, coffee plantations and tropical rainforests, while the coast is full of dive sites.

• From 7 November Virgin Atlantic flies from London Gatwick to San Juan, via Antigua, every Saturday until 24 April 2010 (0844 209 7777; virgin-atlantic.com).

Isla Mujeres, Mexico

According to Mayan legend, this beautiful little island a short ferry ride from Cancún was used for fertility ceremonies. There are some good seafood and fish restaurants and lovely crushed-coral beaches. Day visitors leave around 5pm, when and it gets far calmer – and local shops begin to close.

• Express ferries take 15 minutes from Puerto Juarez and Gran Puerto Cancún (30- and 60-minute taxi rides from the airport) and cost $35 per person. The car ferry from Punta Sam costs $130 per car, $12.50 per driver and $14 per passenger. See isla-mujeres.net.

Ambergris Caye, Belize

It's nowhere near as gritty or edgy as the mainland but this miniature, palm-fronded paradise can be a welcome retreat. Speedboats go from Belize City and the tiny capital, San Pedro Town, has some great guesthouses, as well as cheesy themed resorts. There's not much to do but kick back in a hammock, ride around on a golf cart and take the short boat trip out to one of the world's greatest coral reefs for scuba diving or snorkelling.

• Maya Island Air flies daily from Belize City for $67 return (00 501 223 0734; mayaregional.com). Water taxis to San Pedro run daily for $20 return.