“Where have you gone, Joe DiMaggio?” — the line in Simon and Garfunkel’s 1968 No. 1 hit “Mrs. Robinson” — annoyed the Yankee legend until he understood it.

Paul Simon, a Yankee superfan, told Fran Healy for MSG’s “The Game 365” that the line in the song came to him out of nowhere and that he never expected he would have to explain its meaning to DiMaggio himself.

“I happened to be in a restaurant and there he was,” recalls Simon. “I gathered up my nerve to go over and introduce myself and say, ‘Hi, I’m the guy that wrote “Mrs. Robinson,” ’ and he said ‘Yeah, sit down . . . why’d you say that? I’m here, everyone knows I’m here.’ I said, ‘I don’t mean it that way — I mean, where are these great heroes now?’ He was flattered once he understood that it was meant to be flattering.”

When Simon met Mickey Mantle on “The Dick Cavett Show,” the Mick asked him why he didn’t use his name instead of DiMaggio’s. “Wrong amount of syllables,” Simon told the Yankee star.

Simon continues: “Can you imagine, just the irony of it, I write this line, I don’t know where it comes from, about a ballplayer who I know is really famous but he’s not my favorite guy — and I end up, number one, writing one of the main obituaries . . . and then singing the song in centerfield at Yankee Stadium. I mean, our attachment was so powerful because of that song.”

“The Game 365” airs Monday on MSG+.