Think you there was or might be such a time as this I dreamt of? Idealisation doesn't come close. I have made of Sydney, to which I sailed in 1965, a paradise beyond the powers of fancy. Were I an actor in need of tears I would have only to think of what Sydney Harbour looked like when I first saw it, or how I felt when I left it three years later – not just the place but those I'd grown to love there – for the tears to pour from my eyes like waterfalls.

I was young, I was newly married, my Cambridge degree was still warm in my pocket – a roll of parchment guaranteeing me, I thought, a sort of free ambassadorial passage to any campus of my choosing, and I had chosen Sydney – the world was all before me. Nothing had been good until then: I had hated university, I had been lonely, I craved respect though I had done nothing to deserve any, I had no idea what I was for. But now there could be no doubt: I was on a fourfold mission – to put the past behind me, to enter manhood, to cheer up, to teach Australians how to read English literature… oh, and to overcome seasickness. So that's a fivefold mission.

I failed the last. Some would say I failed the lot. But about seasickness, at least, there can be no argument. We were at sea about a month, and it was a whole year before the ground stayed still beneath my feet. Flotillas of small boats came out to meet us, each stocked to the point of capsizing with touristical junk, of which the most vivid to me still, perhaps because we bought a pair, were carved wooden elephant bookends. In minutes the merchants would ensnare the ship in a tangle of rigging which enabled them to send us up our merchandise, and us to send them down our money. If any more picturesque method of bartering and buying has ever been devised I had not encountered it. Though I had travelled a bit, I had never seen anything like this – a towering vessel roped from funnel to anchor, as though about to be boarded by pirates, so that items of silk and brass, batik and beads, could be hauled up while baskets of paper money and coins were hauled down, all to the accompaniment of a wild cacophony of negotiation and derision. I felt like Marco Polo, surveying the wonders of the world.

When we went ashore it was the same. I was spat at in Port Said – don't ask me why. (By which I mean don't ask me or I'll tell you.) We bought Bob Dylan records for a quarter of their price in Aden, faring better than those who bought boxed shirts only to discover, when they got them back on board, that they had no sleeves or backs. And in Colombo, where I bought my wife a ring from a gold dealer in an upstairs room in a backstreet I wouldn't dare to venture into today, we were chased by a snake charmer who believed we owed him more than we'd paid him for taking his photograph.

But for the lurching sea, everything we saw was lit by the light of a marvellous adventure… well, but for the lurching sea and the Australian stowaway. We'd have helped her more had she not snored when we gave her our cabin floor to sleep on.

We entered the Heads which guard Sydney Harbour late on a broiling February afternoon. The sky was a phosphorescent blue, the air was balmy; in the distance I could make out the arch of the bridge, as full of promise as a rainbow. Bigger and more curious seagulls than I had ever seen in England hovered over us. "Waltzing Matilda" played on the loudspeaker system. I could barely breathe for the kitschy splendour of it all. My wife kissed me. Our new life had begun.

More fun for me, as it was to turn out, than for her.

Everything happened at once. We were met off the boat by the professor who'd hired me. We were to stay with him that night. But before sleep, dinner. The gallon flagons of wine that were passed around the table I will never forget. Claret or riesling – that was your choice in those days. It's my suspicion that wine coming in flagons made one drink more; when you have only red or white to choose from, you concentrate more on quantity than variety. I was drunk quickly anyway, and stayed a man who got drunk quickly for the rest of my sojourn in Australia – a Pom who couldn't hold his liquor. I offer that as partial excuse for my behaviour.

I met the department that night. All there for dinner, to meet the new arrival and to hear the news from Cambridge. Had I, for example, met Germaine before I left? As it happened, I had. It was Germaine Greer – who had lectured at Sydney – I was replacing. A tough act to follow, I was told. So I didn't try. I did it my way – cultivating a leather-jacketed bohemian look, growing my hair, smoking enough cigarettes to kill a hundred men, clicking my fingers like a jazzman when I entered a seminar room, tearing up novels I didn't like, telling students that the novels they liked – works of irrationality such as Wuthering Heights – were rubbish, and otherwise doing all I could not to make the same mistake I'd made in Cambridge, which had been to pass unnoticed through the quadrangles. Stage one of my mission, at least, was accomplished. This time the bastards knew who I was. I had put the past behind me.

It was only half the department which gathered on that first evening – the half that mattered, in my view, since it was the half to which I'd been appointed and which, loosely, shared the literary ideology, which was an ideology of non-ideology, in which I'd been educated. Roughly, our side thought some books were better than others and that it was our duty to teach in that spirit, and the other side didn't. That might not sound sufficient justification for the war that broke out while I was there, dividing the university into rival camps, bringing friendships to an end, ruining the reputations of clever men and making the careers of fools, but it was. For me, the situation was heaven sent. I discovered in myself a taste for bitter controversy, a willingness to inflict verbal blows, a love of violent debate and a peculiar pride in knowing that some people – the wrong people – hated me. It is good for a person who has suffered from acute shyness, as I had, to find that he can cause as much upset as he suffered. Better to be a brute, I thought, than to be a wallflower. And to be thought a brute in Australia, where men were men, and women didn't mind so long as one's brutishness expressed itself in well-chosen language, was very heaven.

It was all heaven. We rented a flat in McMahons Point, overlooking Lavender Bay, from the small balcony of which we could see the Harbour Bridge, the grinning face of Lunar Park, the ferries come and go, and the Opera House slowly rise from nothing. I would walk down in the morning to the jetty – a cliff descent bordered with wildflowers whose scent made me giddy – the sun on my neck, the smell of sea and newness in my nostrils, and think myself transported to another planet. Never have I felt the sheer physical joy of existence – the joy an animal must experience – more vividly. It was a ferry ride of less than 15 minutes to Circular Quay, a 15 minutes I never wanted to end. Other passengers off to work in the city – bank managers in shorts and knee-length socks, lustrous women with the Harbour waters dancing in their eyes – would immerse themselves in the morning papers and not look around them at the wonders of the Harbour, or up, when we passed beneath it, into the engineering marvel of the Bridge. I would never not look around me, I vowed, but of course in time I too sat at the prow of the ferry in the sun and read the papers, especially when they were full of the goings-on in the now infamously riven English department. Funny to think of an English literature department – or to be precise, two English literature departments – being headline news, but we were.

I fell in love, of course. Not just with the Bridge and the harbour it spanned with such unsubtle majesty, but with my colleagues (on the right side of the fence), with their children and their bush houses and their swimming pools, with the cleverness of Australians, with their sense of humour, their affability, their sentimentality and their recklessness, with the friends I made and, inevitably, with a student. I was too stimulated not to fall in love. I was a boy from a cold northern city who had been educated in a cold East Anglian town. Anywhere with a glimmer of light and a ray of sunshine would have turned my head, but Sydney was glamorous – growing into a great and dangerous metropolis, but still exquisitely poised between that and a gentle, overgrown country town populated with Greek and Italian market-gardeners. In half an hour from where we lived we could be on a harbour beach or an ocean beach, washing scallops down with riesling, watching lovers kiss and surfers tumble and children play as though Sydney were a giant kindergarten without a worry. No worries. The phrase originated here. And on top of this I was enjoying fame as a young Turk lecturer saying outrageous things and enjoying pouring fuel on an already inflammable institution. How could I not fall in love?

You fall in love differently when you are young and far from home in a seductive place. You fall in love with the very air you breathe, and the vivid colours and the unbearably sweet sensation of distance and unaccustomedness. If the person already embodies the spirit of the place that has seduced you – its beauty, its liveliness, its quickness, and somehow its faraway sadness, for Sydney always struck me as melancholy – and if your trysts are indistinguishable from this landscape, the city as much the site of passion as your heart, you fall heavily. There were strong arguments all round for not allowing this to happen – though age difference was not one of them: I was scarcely older than my students – but happen it did. To speak of ruination is to put it too dramatically, but in the end all parties to this affair suffered keenly, me included, though I was the least entitled to sympathy.

Against these continuing aches of love and falsity my life at Sydney went its way; the arguments at the university grew more bitter, my friendships with Australian men more devoted. Last year the best of my Australian friends of this era died quite suddenly. Terry Collits, a literary man, a product of the Christian Brothers, a manly Marxist theoriser, a consummate clown, a cricketer and for me the very voice of Sydney in the quick alternations between seriousness and wild mirth, between modernity and old-fashionedness – he wore the most terrible, office clerk's trousers – between confrontation and sentimentality. We fought and betrayed each other, we threw wine at each other – I wanted to finish off those trousers – and we wept in each other's arms. Friendships with Australian men almost invariably took this form, always on the verge of tears and violence. The heat had something to do with it, and those flagons of claret and riesling; but it's my theory that this intensity of mateship goes back to the earliest days of the white men in Australia, before the women turned up, when they had only themselves to play with.

Clever bastards I found them, too. More knowledgeable than I was about everything, liberating themselves from the intellectual patronage of Poms like me by being altogether more cultivated. So back we'd go from a morning playing squash to an afternoon listening to Brahms's clarinet quintet and then on to the balcony of Terry's house in the fading sunshine for chess and the wine coming out. I think of him as Sydney, a Sydney now unbearably diminished for me.

So why, since I loved IT so much, did I leave less than three years after my rhapsodic arrival? The simple answer is that you can have too much of a good thing: there was too much fun and not enough work, Sydney had become a sort of Lotus Land to me, a seduction of the senses, and I was, after all, an Englishman with English ambitions to which the sunshine and the wine and the overall beauty of everything and everyone were at last inimical. It's a sad but necessary fact for some of us to face: that we fare better when the going isn't easy.

But that doesn't explain it all. The splitting asunder of the English department played a part. Our side had lost the fight if not the argument, and now many of my closest colleagues were leaving, dispirited. And I, too, was splitting apart, unable to make mature choices between love and responsibility, or harder still, between love and love. So you could say the decision to return was as much an act of indecisive flight as anything else.

The fun was over anyway. As was the last remnant of what could be called my boyhood. Job done, in that case. On the day before I sailed away, I sat on a bench in a park overlooking McMahons Point – the harbour seen from the other end to the one we entered by – and shed tears enough to float a paper boat on. I felt like Matthew Arnold's Forsaken Merman, gazing at the sleeping town from the sand hills, singing: "There dwells a loved one," though I was the one doing the forsaking.

I have been back to Sydney many times since – once I even rented a house and stayed there for a year, trying to reintegrate it into my present life – but I can't trust me in the place. It's too like visiting myself as I was, as I might have been, as I had no business being, as I am glad I am not any more, allowing that one is bound to be ambivalent about one's younger self. It's also a bit of a burden on Sydney that it has to bear so much. But then again, it flaunted its beauty, so can it be all my fault that I fell for it?

Born in Manchester in 1942, Howard Jacobson is a novelist and broadcaster. His latest novel, The Finkler Question, is published on 2 August at £12.99