From the summer of 1996 until he died in July 1999, I worked for John Kennedy Jr at his monthly glossy magazine, George. ‘There’s no one like you, Inigo,’ he said on the phone when he offered me the job. I was always going to take the post if he wanted to give it to me, but he had a way of never making it easy for anyone to say no.

George’s offices were on the 41st floor of a skyscraper on Broadway and West 51st – it was one of many titles run by Hachette Filipacchi Media – and the first desk I had was at a south-facing window, looking down the Hudson River. You could spend a lot of time with that view, especially when the big boats came in, as they did just before New York’s Fleet Week when an aircraft carrier came to town. From where I was sitting, each one dwarfed the World Trade Center.

My immediate neigbour to the left at that desk was the editor of another Hachette title, Car and Driver, or ‘Crash and Bumper’, as I used to think of it. To my right were John and his assistant, Rose Marie. ‘This is not a good time of year for John,’ she told me in November 1996 as the anniversary of his father’s assassination approached. With the Thanksgiving holiday, the news in Washington typically seized up, so TV channels and weekly magazines would turn to the archives for the ‘latest’ on the death of JFK. This happened year in year out, until 2001 when one American Day of the Dead was usurped by another.

If for some people 22 November wasn’t the best of times, there were others who couldn’t help but try to make it worse. One company executive suggested that John should interview Marina Oswald for the magazine.

John talked about his father only once when I worked for him, and that was in Cuba. ‘Are you much the same height as your father?’ Fidel Castro asked him soon after the two of them met in October 1997, during the 35th anniversary of the Cuban Missile Crisis. He and I, along with the photographer Herb Ritts and his assistant Matthew, had gone to Havana with the hope of getting an interview with Castro. The interview never materialised but we did have dinner with him, in a place built by the Americans in the 1950s that looked as if it could have been part of the campus of the University of East Anglia.

A few hours earlier we’d been 175 miles away in Santa Clara, at Che’s tomb, and had given up hope of meeting Castro. For four days, we’d travelled around Havana and western Cuba in a minivan, Herb Ritts with his rice crackers and endless talk of giving Castro a copy of his massive and massively homoerotic book of photographs, Work. By the end of the journey, the crumbs of Herb’s rice crackers were everywhere. ‘When do you think I should give Fidel my book, John?’ Herb kept asking, until I pointed out that Castro wasn’t wildly enthusiastic about gay life. ‘What about putting it in the post,’ I said.

We went to the Bay of Pigs, where our minders encouraged John to have a swim; he met the suggestion with the same sort of silence I imagine he gave the executive who suggested he interview Lee Harvey Oswald’s widow. We walked about Santa Clara, no one apparently knowing who John Kennedy was, and he loved the anonymity (he was planning to go there again, on a real holiday, when he died). It was striking when people walked passed him without even bothering to look him. He was not unstriking. And then in mid-afternoon on the penultimate day a call came through; the man from the Foreign Ministry said it would be a good idea if we were back in Havana that evening. For much of the journey we were the only vehicle on the road. John said that the prospect of interviewing Castro was like an anvil over his head. Then he opened the window and put that head out into the breeze.

When we reached the hotel we were staying at there was another call. There was to be a supper: be ready at nine. No interview, and no photographer. ‘Do you want to take my book, John?’ Herb asked. The man from the Foreign Ministry appeared at 8.30. The lift went down to the basement and we walked through the kitchens to the underground car park, where the same minivan was, the rice cracker crumbs all about the seats, the curtains now closed. Whether this was to stop people looking in or to prevent us from looking out, who knows. It was so dark in Havana in 1997, it was the darkest city at night I’d been to; there were few street lights and cars only had one headlamp. We couldn’t have seen anything if we’d wanted to.

‘He was a little taller,’ John said evasively in reply to Castro’s question, ‘a little thinner.’ John was about to turn three when his father was killed. This was at the drinks stage of the evening. The other people there were Ricardo Alarcón, then head of the National Assembly; José Ramón Fernández, the man who’d repelled the invasion at the Bay of Pigs; Juanita Vera, Castro’s translator; the foreign secretary and Castro’s ADC. A tray of 1950s cocktails went about the room; I remember the glistening condensation on the glasses more than the contents. None of the Cubans apart from Castro and Vera said a word. He talked about what he was reading: Churchill, Zweig. ‘What did you think of Nixon?’ he asked John. Nixon had invited the young Kennedys to the White House in 1969; Tricky Dicky had been gracious and kind.

Those were just the opening moments; what followed was a five-hour, five-course dinner at a broad table covered with an embroided cloth. John sat opposite Castro; I was on the president’s right.

Only at the end of the evening did Castro return to John’s father. As we were leaving the dining-room, Castro stopped and said: ‘You know Lee Harvey Oswald was trying to get to Cuba.’ Oswald had been refused a visa at the Cuban embassy in Mexico City a few weeks before the assassination. John nodded, and Castro walked on before stopping again. ‘You know it was hard to allow Americans into Cuba at that time,’ he said. John nodded again. I think this was Castro’s way of saying that he hadn’t had anything to with the assassination, but that if he had allowed Americans into Cuba in November 1963 then Oswald wouldn’t have been in Dallas on the 22nd. Mind you, and who’s to say, maybe there would have been another gunman at Dealey Plaza.We said goodbye and went back to the hotel.

The next day we flew back to New York via Cancun. We’d been given a box of cigars each, both wrapped in bright pink wrapping paper. If the idea was that the paper would disguise the contents of the boxes then it failed completely: the words ‘Cuban Cigars’ might as well have been stamped on them. It wouldn’t look marvellous if John was stopped at an American airport with Cuban cigars so I agreed to put them in my bag.

‘Hey, John,’ said an officer from the Immigration and Naturalisation Service who’d never met him before. He swung open a gate and let us through, no need for passports. ‘See you again,’ the agent said. I handed John his pink-wrapped box: who was going to stop that Kennedy? As I said, he had a way of making it difficult for anyone to say no.