Bob Dylan is 70 on Tuesday. If like me you are a Dylan fan, you will have been waiting for this event for months, perhaps even years. If you are not a Dylan fan – and I realise there are some who have not seen the light – you'll wonder what all the fuss is about. Even those happy to hum along to his early hits may now think his life is but a joke.

I am a Dylan fan, but not quite a fanatic. I'm not one of those people who goes to every gig, collates setlists, chats with other obsessives on internet messageboards, or criss-crosses America in an RV in pursuit of him. I first saw him in 1978, when he played a week of concerts at Earls Court. He hadn't performed in the UK for nine years, and these were huge events. For me, just finishing university, this wasn't a show, it was a rite of passage, a communion, a consummation. I have kept my programme and even my ticket from that 1978 concert. I've just discovered that ticket stubs from those concerts are selling online for £20 – three times what I paid for my balcony seat back then – but I'm not selling. For me, this concert was a watershed. I had found someone in whom I believed totally.

What's odd is that I've never questioned that faith, even when his inspiration flagged. As a recent poll of leading musicians in Rolling Stone magazine suggests, all his signature songs are from the first 15 years of his career. Blood on the Tracks in 1975 marks the end of that period of unquestioned greatness. Desire – the first album I got to know well thanks to John Peel playing it complete on his late-night Radio 1 show as soon as it was released in 1976 – retains some of that aura and ambition. But Street Legal, released in 1978, shows a marked falling off. Music writer and Dylan specialist Greil Marcus immediately recognised its inauthenticity. The banal, mechanical rhyming underlines the diminishing energy, and not even the brilliant Señor can save it. Dylan's muddled middle period was beginning.

It is tempting to conjure up a brilliantly revisionist argument – that the true glory of Dylan resides in the mid-80s albums Empire Burlesque and Knocked Out Loaded, for instance – but it can't be done. But the albums in the middle period should not be completely dismissed, even though many critics more or less gave up on him in the 80s. Some artists who produced great work in the 60s and early 70s – Paul McCartney and the Rolling Stones spring to mind – really have added nothing of note to their oeuvre since. They remind you of those young playwrights who pile out great plays before realising just how difficult it is. Once they've lost that innocence, they either dry up, or, worse, spend 40 years repeating themselves.

Dylan had lost the knack of producing great albums – he was blocked creatively for much of the 80s – but could still knock out the occasional great song. Shot of Love, released in 1981 at the height of his religious phase, ends with the poignant Every Grain of Sand. Infidels, from 1983, has Jokerman. Even Down in the Groove in 1988, reckoned by some to be his worst album, a bare 30 minutes of insipid new songs and uninspired covers, has one song I really like – Death Is Not the End, which has one of those gloriously mournful Dylan harmonica intros. This is enough to forgive him a great deal, even the peculiar version of Shenandoah on the same disc.

Something was clearly amiss in the 80s – mid-life crisis, too much touring, personal problems, who knows? His gift was always instinctive rather than entirely controlled. No one who could publish Tarantula, his rambling 1966 sub-Ginsbergian poem, or waste his time on the tedious and incomprehensible mid-70s film Renaldo and Clara could claim to have impeccable artistic judgment. When it was easy for him, it was too easy; when it got hard, maybe he panicked.

You certainly sense a degree of panic in his memoir Chronicles, published to great acclaim in 2004. "I hadn't actually disappeared from the scene," he writes, "but the road had narrowed … I was lingering out on the pavement. There was a missing person inside of myself and I needed to find him … I felt done for, an empty burned-out wreck. Too much static in my head and I couldn't dump the stuff. Wherever I am, I'm a 60s troubadour, a folk-rock relic, a wordsmith from bygone days, a fictitious head of state from a place nobody knows. I'm in the bottomless pit of cultural oblivion." This is laying it on a bit thick. Dylan, for all his virtues, never quite knows when to stop when it comes to word production.

Dylan's audience diminished in the 80s. It was easy to get tickets for gigs, and you could see him in smaller venues. But we true fans never wavered. I would have bought the albums even if they hadn't still contained the occasional gem. I'd signed up for the religion at Earls Court in 1978, and there was no going back.

He wasn't confined to small venues for long. The official bootlegs won back the critics, and gave fans like me new cause for fascination. Then came the run of albums, beginning with Time Out of Mind in 1997, that suggested his gifts had returned, albeit in different form. He seemed to have found his voice again as he ruminated on mortality and communed with the ghosts of the great bluesmen.

Late Dylan is fascinating: the darkness, the obsession with time draining away, the refusal to stop touring even with a voice as rough as sandpaper. He transcends criticism now. When he makes a Christmas album, as he did in 2009, we nod sagely and add it to our collections, marking it down as an homage to Bing Crosby, one of his earliest heroes. I could probably live without it, but I'm not embarrassed to have it in my collection. Indeed, his croaky rendition of O Come All Ye Faithful (with verses in Latin and what may be English) may turn into a regular part of my Christmas ritual, like eating too many cheese footballs before dinner.

I bought an album recently to fill a gap in my collection: Bob Dylan (A Fool Such As I). It was released in 1973 by Columbia when Dylan announced he was moving to Asylum. Generally seen as a malign attempt at subversion from his long-time label, it is filled with covers he had recorded but never intended to release – Can't Help Falling in Love, Big Yellow Taxi, A Fool Such As I, Mr Bojangles. Critics mocked it, and it was quickly deleted when Dylan made his way back to Columbia. It's expensive to buy – the CD cost £55 from a seller in Germany. I'm listening to it now. And do you know what? It's great, money well spent; the sabotage didn't work.

Dylan has been omnipresent for the past 50 years, yet we know next to nothing about him. Fat books pour forth, especially in this anniversary year, yet he still eludes us, this rolling stone, this balladic thin man. Todd Haynes's clever, beautiful, moving film, I'm Not There, is a perfect summation of Dylan's career, because he truly does not seem to have been there during those 50 years. The six Dylan personas incarnated by six different actors (including a black boy and a white woman) perfectly represent Dylan's elusiveness, his partly deliberate evasiveness, his stolid evanescence (the lyrical logorrhoea is catching).

I have a framed photograph of the young Dylan on the wall next to my desk. He is thin, wearing jeans and a check shirt, looking straight at the camera with a hint of arrogance, hands in pockets, his guitar case sitting on the road beside him. He is standing outside, in what looks like an empty car park, surrounded by pools of water. He is alone, self contained, at one with himself in this alien landscape. He has a slight smile, as if he has some secret information. Yet he never spells it out, never makes it easy for us. That may be why the love affair endures; the mystery remains. The answer is still blowin' in the wind.

In 2005, the Guardian asked me to review a Dylan gig. This was probably a mistake as the chance of an objective assessment was nil. The reviews editor may have realised this when I tried to give the show seven stars. I remember becoming tearful during Visions of Johanna, one of his truly great songs, which even the ultra-reductive late Dylan is incapable of reducing to rubble. A man standing beside me saw me making notes through the tears. "We're just crossing the ocean with Bob," he said. "Write that down." And I did, because he had summed up what it means to be a Dylan fan. We are on a voyage, and the voyage never ends.