The actor says the King of Pop would disguise his voice in the calls

When Michael Jackson calls, you pick up the phone. At least that’s what the King of Pop hoped would happen when he would place frequent prank calls to Oscar winner Russell Crowe.

“It’s funny,” Crowe fondly recalls of late music legend’s antics. “Most of the time Michael would be speaking in a rough voice,” the actor tells PEOPLE. “[He’d say], ‘This is the hotel, you have to get out,’ and I’d say, ‘Who is this?’ and he’d say, ‘Oh don’t worry, Russell, it’s Michael!’ He [told me] straightaway. I didn’t have to guess who it was or what was going on because he always wanted to have a chat afterwards.”

Get push notifications with news, features and more.

Says Crowe of Jackson, who died in 2009, “I’ve always told that story with great affection because I’ve always felt that he must have recognized something in me that trusted I wouldn’t be bothered by it.”

Crowe, 51, next stars in and directs The Water Diviner, out April 24.