Alan Titchmarsh's second novel, The Last Lighthouse Keeper, which is published in paperback tomorrow on his 51st birthday, has slightly fewer sex scenes in it than the first, a fact that has caused some consternation in middle England. "You get ladies coming up to you in bookshops, what you might call nice, good, sparky ladies, saying about Lighthouse, 'Not as much in it as Mr MacGregor then ...'

I had one last week. I said, 'Did you feel a bit short-changed?' 'No,' she said, 'But I did enjoy them in the first one.' " Titchmarsh, who is sipping Perrier in a London hotel, exaggerates his Yorkshire accent when he says this, which is interesting as his mother, in her 70s and still living at Ilkley, is not one of these ladies. "I said, 'Do you mind now?' because she hadn't spoken about the books, and she said, 'As long as there's no more licking.' "

It does all seem to boil down to sex with Titchmarsh. Never before has gardening been so erotic. He does his best on Gardeners' World and Ground Force with his decking and railway sleepers, but there's Charlie Dimmock, bra-less on the end of her spade. (Unresolved Sexual Tension? "I hope so," he says.) And there he is, a piece of Reader's Digest crumpet, telling you what to do with your flowering cherry, a man not afraid to bury his arm in a herbaceous border.

He's unthreatening, perennially perky; a cheery glint in his eye, a strong, earth-ingrained hand on your arm. He calls TV "the box", talks about "folk", his strongest swearword is "crumbs". He's always being voted something of the year - Gardener of the Year, Yorkshireman of the Year, TV Broadcaster of the Year. Presumably those nice, sparky ladies tick his box.

He also came second in last year's Literary Review Bad Sex Award, for a passage in Mr MacGregor about a TV gardener with a fatal sexual magnetism, in which there is a lot of "slipping" and "sliding" and hands running down bodies until they "find what they're looking for" and, at one point, a tidal wave of quivering carnality theatens to engulf somebody. He was runner-up to Sebastian Faulks, who didn't turn up. "When I see him I must apologise to him for having stolen his thunder. I stood and said one line - 'I'm glad I came and did so well in the face of such stiff opposition.' And I stood down."

A good sport, then, Alan Titchmarsh, professionally cheerful in the face of a good ribbing, maintaining a fixed smile even when asked on Parkinson a couple of weeks ago to read out the rude bits. Even when fellow guests Cindy Crawford and Ronan Keating pass the book around and read out some more. But inside?

He shifts uncomfortably and frowns. He's neater than he is on the telly, his red and white gingham shirt politely folded over his red pully. His hair is newly cut, in boyish tufts. His hands are surprisingly small. They're clean, but dry and scaly.

"What's upsetting," he says, chewing his lip, "is that it's not fair. People quote what they call 'sex scenes' out of context, they take the mickey, they think, 'Does he know how this is done?' Yeah, I do, I've got two children [teenage girls]. But these aren't 'sex scenes', they're love scenes. They're not sordid. They're tender. I was desperate to show that love-making can be a two-way thing, that we're not all bastards. Men tend to write about macho men. I wanted to write heroes who are gentle, not wet - real men but with a sensitivity that women would find attractive."

How did he cope with the Parky thing? "The book sold out the following day," he said grimly. "So I smiled."

Titchmarsh's real but sensitive nature, gentle, not wet, has made the son of a plumber who left Ilkley county secondary school at 15 with one O-level in art phenomenally successful. His two novels sold 70,000 and 80,000 in hardback. The next, Animal Instincts, will be out in the autumn. He has just signed a £1m deal with BBC books for three new gardening titles (he's written about 30 already). He has columns in the Express and Radio Times and Gardeners' World magazine. Gardeners' World and Ground Force regularly top the BBC's ratings. He must be raking it in. ("Don't forget the agent's commission and the income tax," he says, "but it is nice".) He has a house in Hampshire, with an acre and a half of garden, 30 acres of woodland, and a specially built theatre where he and his wife, Alison, whom he met in amateur dramatics, perform skits for their friends. There is also a 38ft boat, in which he pootles about the Solent, and a flat on the Isle of Wight.

They bought the flat last September. It was to be their bolt hole, their escape from the constant "Can you help with my aphids" that his fame has thrust on him. But then there was some bother with a tree and now the whole world knows about it. "It was a sycamore," he explains wearily. "Some sycamores are lovely but this one wasn't. And yes it was a sea view flat and half the sea view wasn't there. It was just me putting me hand up in class, saying, 'would it be all right if we?' and then it was all this and that and people saying ... In the end it was diseased and had to come down anyway, so poetic justice."

He grits his teeth. He doesn't look so sensitive for a moment. "We planted four acres of trees in Hampshire. You don't read about that in the papers. Just the sawing down of one sycamore." He scratches his ankle. "It's tricky getting away."

Talking to Titchmarsh, you get a strong sense of how he sees himself. There is not just the considerate lover, but the loner, the romantic. The new novel is about a lighthouse keeper who dreams of sailing around Britain. Titchmarsh has dreams of leaving, too. "Of escaping." (His own boat is a motorised trawler. It's not quite sailing, he admits, more "driving on the water"). The lighthouse keeper's wife was killed in a car crash. There are tears in Titchmarsh's eyes when he reminds me of a passage. "The bit where he realises Amy [the new woman in his life] means something to him and he looks out to sea, 'please let me go', that terrible feeling that she wouldn't let go of him, 'you are mine' and he still felt that ... and how could he love two people?" He puts his thumb in the corner of his eye. "I hope I did that bit sensitively."

Then there is his status as a TV personality. He talks quite a lot about his fame which began when he was a presenter of Pebble Mill (his first TV appearance was on Nationwide in 1979, "greenfly invading Margate"). He didn't ask for it, he says. He wants you to know he's just an ordinary chap - "first bloomin' dry day and here I am nattering on about myself"; "Hark at me going on and on" - while not allowing a chink in his armour, or his waterproofs, to show. Some people have suggested that Nelson Mandela did not look so happy to have his garden done over. "No, no. That was good humour," he says crisply. "No, no. He just kept walking around saying, 'Gee whiz'." Others have drawn attention to the fact that Titchmarsh doesn't tease out his roots or that he overdoes pebbles. "I can't be doing with people being grumpy," he says grumpily. "There isn't a right or wrong. There's my way and there's your way, and as long as they grow ... Stop it. It's self-expression that counts."

And then there's a little misunderstanding about Dimmock. He's telling me how kids like Ground Force, how mums come up and tell him about their "rampant lads of eight, nine, 10" who are leaping around for Top of the Pops and then, "Whomph, the theme tune comes on and they're sat on the floor and they're silent." Perhaps that's the draw of Charlie, I suggest. "That's the other thing," he says crisply. "She's a mate, but the female population isn't watching it for Charlie and the female population makes up 60% of the audience, so it's not just for her." Perhaps a nerve has been twanged.

Most of all, though, Titchmarsh is a man of the elements. "You can't control the weather," he says, and he bends forward to whisper his secret, "and I like that." He talks about the day before, when he'd been shut indoors writing and getting irritable, and then went and cut the grass and felt much better. Bill, who's full time in his garden, told him, "You always do."

He couldn't live in town, he says, "I'd curl up and wither away." He's not, repeat not, a TV personality. "I find myself looking in Hello! and OK! and thinking they inhabit a different world to me. I was looking the other day and there were these parties and soirees and little sorties that are going on. I'm not a professional showbiz icon. I'm me. I don't want to be decorative. No, no, I'm not filling your pages for you, smiling in a nice suit. No, that's boring."

He's very convincing until Terry Wogan walks in and waves wildly at him across the room. "Saw you at the Ivy the other night," he hollers. "Did you see Donald Sutherland? Got up and left." Titchmarsh smiles wanly. "I do like the Ivy," he says apologetically. "I don't stand out. I'm nobody there."

Then he turns back to Wogan. "Apparently, it was too smoky for him," he calls.