‘The peninsula carved by Time out of a single stone,” Thomas Hardy called it. He set his strange 1890s novel The Well-Beloved on Dorset’s Isle of Portland, which stretches out “like the head of a bird” into the channel. Connected to the mainland by a causeway and long, pebbly Chesil Beach, the “island” is famous for the pale limestone that built Saint Paul’s cathedral. There are still quarries here, and striking stone structures: churches and castles, harbours and prisons.

The three of us arrive in Weymouth, a couple of miles north of Portland, from different directions. My mum (using her bus pass) along the coast, my brother by rail from Somerset, and me from Essex (via London). The Waterloo to Weymouth train takes three hours. South Western Railway has a patchy record, but today’s journey, with air-con, window seat, table and power sockets, is a model of civilised travel. The onboard wi-fi comes with free entertainment, from Love Island to Darkest Hour; the scenery offers marshy estuaries and miles of brackeny New Forest.

We meet at King’s Statue on the seafront. George III, in a blue and ermine cloak, looks anxiously through palm trees towards a replica of his royal bathing machine. Regular buses head from here towards Poole and Axminster along the Jurassic Coast. We take the open-topped 501, which leaves hourly in summer for the red-and-white lighthouse at Portland Bill on the far end of the peninsula.

Facebook Twitter Pinterest The open-topped bus collects day-trippers at sunny Portland Bill, ready for the scenic return journey to Weymouth. Photograph: Stuart Fretwell/Alamy

After generous crab sarnies (£10.25) at Portland Bill’s The Lobster Pot café, we set off along the coast path on the hottest day of the year so far. The heat draws lightning-fast lizards out on to the sun-warmed rocks and releases the Mediterranean aroma of wild marjoram and aniseedy fennel into the shimmering air. Three shadeless miles to Church Ope Cove leave us sweaty and we swim off the white rocky beach, cooling down in the clear turquoise water.

The ruins of 12th-century Rufus Castle stand on the cliffs above us, near Portland Museum (adult £3.50, child 6-15 £1). Family planning pioneer Marie Stopes founded the museum in 1930. She was friends with Hardy, who based Avice Caro’s house in The Well-Beloved on one of the museum’s thatched stone cottages. The rooms are stuffed with local curios, from mummified cats and puffer fish to stone masons’ tools and a U-boat compass. There’s more in the shady garden – fossilised dinosaur vertebrae, a Roman sarcophagus, the wooden arm of an old windmill…

Two of Portland’s most interesting churches are close by: ruined St Andrew’s, its overgrown graves carved with piratical skulls and crossbones, is five minutes’ walk away down a wooded path, where butterflies flicker among the buddleias and clematis. Clifftop Georgian St George’s, 15 minutes in the other direction, has an evocative time-warp interior of box pews and twin pulpits. Nikolaus Pevsner called it Dorset’s most impressive 18th-century church.

The White Stones Art Café in the nearby village of Easton supplies cake and huge mugs of tea in a vine-covered courtyard full of mosaic archways and blooming agapanthus. From the stop outside the door, the year-round number 1 bus, which shuttles between Portland’s villages and Weymouth every 12 minutes, takes us down to Portland youth hostel, where we’re staying the night.

Facebook Twitter Pinterest White Stones Art Cafe

The sturdy brick hostel overlooks neat houses built for the sailors competing in at the 2012 Olympics. For dinner, we have seafood at the sunset-facing Cove House Inn (mains from £8.50) on Chesil Beach, down the road. We stroll back next morning and follow the coast path to Quiddles for breakfast. This cheerful beachside cafe in a candy-pink-painted former public convenience serves sustaining fry-ups (from £5.50) or veggie-friendly bubble and squeak (£3). This sets us up for the short, steep climb up to Tout Quarry (admission free), half a mile away at the top of the cliff. This labyrinth of hidden carvings and half-emerged stone creatures winds through banks of summer flowers. In one disused pit, Antony Gormley’s carved figure, Still Falling, dives headfirst down the cliff.

The bus takes us down again to Portland Castle, built by Henry VIII. While the others check out the waterfront, I climb the old Merchant’s Incline, once a railway taking quarried stone down to the harbour. At the top, I walk through the daunting archway of 19th-century Verne Citadel, now HMP The Verne, to visit the Jailhouse cafe (panini from £5.25, jacket potato from £6), which reopened earlier this year. It’s an optimistic place with one of Portland’s best views, over miles of sea and curving breakwaters to the chalk-cut white horse at Osmington.

Facebook Twitter Pinterest That’s the way to do it … the Punch and Judy show on Weymouth beach. Photograph: Matt Cardy/Getty Images

Up to 52 passengers and their dogs can sardine into the old wooden ferry for a boat ride back to Weymouth. The 1930s boat was requisitioned during the second world war to carry soldiers out to the breakwater forts. Now it’s a pleasure boat again and we relax as sea air soothes our sunburned faces.

After the wild cliffs and lonely quarries, Weymouth’s screaming fairgrounds and beeping arcades feel more than usually crowded, but there’s still space on the long sandy beach for a last swim. Hardy regularly swam here. In The Dynasts, his closet drama in verse, he has George III telling Pitt the Younger “The bathing is unmatched … The air like liquid life.”

We sit by the Victorian kiosk near the Punch and Judy show and drink tea before heading to the station, looking out at the view described by Hardy: white cliffs, a cannon-topped headland and “the crescent-curved expanse of the Bay as a sheet of brilliant translucent green”.

• Train travel was provided by South Western Railway (London to Weymouth from £14.50 one-way); bus travel was provided by First and ferry trips by Coastline Cruises (£7 one way). Accommodation was provided by YHA Portland (dorm beds from £13, private four-bed rooms from £39)

Looking for a holiday with a difference? Browse Guardian Holidays to see a range of fantastic trips





This article contains affiliate links, which means we may earn a small commission if a reader clicks through and makes a purchase. All our journalism is independent and is in no way influenced by any advertiser or commercial initiative. By clicking on an affiliate link, you accept that third-party cookies will be set. More information.