Sunday afternoon, Winnifred beach, Jamaica. Archie, our nine-year-old, is off snorkelling on the coral reef with a group of local lads who are teaching him to catch angelfish with his bare hands; George, nearly two, is playing with the kids of the friendly beach shack owner. She is feeding us jerk chicken and chilled Red Stripe while we relax under a tree.

There's a game of football going on behind us; reggae music and the delicious smoky aroma of jerked meat are floating on the breeze. The sun is out, the sand is powdery white, the sea that unmistakable Caribbean blue and, contrary to popular belief, no one is trying to sell us ganja. It is the perfect family holiday moment.

Alas, moments like this are quite rare in Jamaica, at least ones where you hang out with locals on a public beach. There are only 87 recreational public beaches; guesthouse and villa complexes and powerful all-inclusive resorts which dominate the island's tourist industry control access to 336. The only resident Jamaicans you'll see on most of them are cocktail waiters (whom you are not allowed to tip).

Even Winnifred, one of Jamaica's most famous family beaches, is under threat. The local community is embroiled in a four-year legal battle with a government agency, which, if it wins the case, plans to change the right of access in favour of a hotel developer. Although judging by people I spoke to, the Free Winnifred (free-winnifred.com) campaigners have formed an effective lobby and are confident of victory, or at least a favourable settlement.

Fast friends … George and chums on Winnifred beach. Photograph: Gavin McOwan

Of course, the government is pursuing this policy because what most foreign tourists come to Jamaica for is to sit on gorgeous tropical beaches in fenced-off resorts, safe in the knowledge they need have little or no contact with the local population.

One friend back home had even said: "Why would you take your kids to a country full of drugs and gangsters?" To which my answer is, I didn't.

Avoid the capital, Kingston, and the main resort areas of Montego Bay, Negril and Ocho Rios, and you'll come across very little crime and few tourist hustlers. We spent a week in Portland, in the island's north-east, and Jamaica felt like a sleepy, friendly backwater. One proud local told me, "This is the safest place in the Caribbean."

That may be over-egging it, but it is certainly one of the safest parts of Jamaica, as well as the most rugged, and perhaps most beautiful spot on an island spoilt for dramatic scenery. In Portland the lush rainforest tumbles all the way from Blue Mountains to rocky coves and turquoise sea.

A 10-minute stroll from where we were staying – Goblin Hill Villas, just east of the parish capital Port Antonio – is the deep inlet known as the Blue Lagoon, where Brooke Shields cavorted to fame in the eponymous 1980 film. (The tourist board renamed the place the Blue Lagoon in the wake of the film's success: it was originally known as the Blue Hole.) There were a few guys peddling tourist tat and offering boat rides, but they left us in peace to swim in the impossibly blue water. I felt pretty pleased with myself for buying a couple of "fresh" lobsters from one of them, but when we got back to the villa, our housekeeper, Annie, scolded me for buying dead ones.

Goblin Hill is in an idyllic setting overlooking San San beach and surrounded by dense forest filled with birds. There's also a nature trail and acres of grass for kids to run around on. The spacious villas are open-plan, with patios and two large en-suite bedrooms upstairs, both with spectacular sea views. They must have been the height of sophistication when they were built in the 1970s, like the set of Abigail's Party transported to the tropics.

Sadly, Annie's menu (under instructions from management) was also stuck in the 1970s, featuring packet soup, pancake mix, even bottled lime juice and instant curry ... in Jamaica. Not that the kids minded: they wolfed her meals down quicker than any of the "proper" food they get at home.

The whole slightly complicated housekeeper system (each villa comes with one) is designed for wealthy Jamaicans, but for self-conscious Brits having a person to wait on you all day was a protocol minefield with a slight undertone of colonialism. Luckily, Annie was prudent enough to let on what we were supposed to tell her to do for us. She was also fantastic with the kids, and after a few days of waking up to an alfresco breakfast of fresh fruit and delicious ackee and saltfish (in Jamaica it is punishable by law to serve that out of a packet) I realised I could have got used to it.

Oarsmen on the Rio Grande. Photograph: Gavin McOwan

One of the highlights of Portland is a ride on a banana boat. No, not the yellow plastic ones that bomb around tourist shores the world over, but the long elegant bamboo rafts that were once used to transport bananas from plantations to port (Rio Grande Tours, +1 876 993 5778, $75 per raft). Errol Flynn, Portland's most famous former resident, popularised the idea of being punted downstream by local boatmen.

When we arrived at the starting point upstream, the sun was so intense I feared the kids would melt, but the moment we nudged away from the bank, a breeze blew up and we dipped our feet in the cool water running off the Blue Mountains. It was such a serene journey – the raft gliding down a wide pebbly river walled by towering bamboo – that even the youngsters sat perfectly still for the two-hour ride.

It was another fantastic Jamaican day, but the island saved the best until last. Frenchman's Cove is a tiny beach that makes a huge impression. The small stretch of pure white sand is framed perfectly between cliffs peppered with trees and creepers. There are waves to play in, and a small river the colour of jade snakes into the sea down one side of the cove.

It is an intoxicating place, all dense tropical vegetation and water in every hue of blue and green. And there wasn't another soul there (perhaps because it was Monday morning and there is a $5 entrance fee). We spent hours swinging from the Tarzan rope and dropping into the warm river below, then waded to the other side, where we discovered a small beach hidden by the huge leaves of overhanging trees and created a Robinson Crusoe-like den.

We didn't need watersports or pedalos or kids' clubs. It felt like we had Eden all to ourselves.

• Virgin Atlantic (0844 209 7777, virgin-atlantic.com) flies from Gatwick to Kingston from £519 rtn inc taxes.Large two bedroom villas at Goblin Hill (+1 876 925-8108, goblinhillvillas.com) from US$220 (around £140) per night, including the services of a housekeeper. For further information see visitjamaica.com