Back in 1961, a record exec named Lou Levy took Bob Dylan to Jack Dempsey’s restaurant on 58th and Broadway in New York to introduce the folk musician to the legendary boxer. Dylan recalls the encounter in his 2004 autobiography, Chronicles, Volume One.

Dempsey sized up the young man and shook his fist at him. “You look too light for a heavyweight kid, you’ll have to put on a few pounds. You’re gonna to have to dress a little finer, look a little sharper—not that you’ll need much in the way of clothes when you’re in the ring—don’t be afraid of hitting somebody too hard.”

“He’s not a boxer, Jack,” Levy corrected the champ. “He’s a songwriter and we’ll be publishing his songs.”

“Oh, yeah, well I hope to hear ‘em some of these days,” Dempsey replied. “Good luck to you, kid.”

The advice wasn’t quite as useless as Levy thought at the time. Dylan actually was a bit of a boxer in his spare time, and although the sartorial tips clearly didn’t stick, the part of about not being afraid to hit too hard has served him well in the half century since.

Dylan, born Robert Zimmerman, began boxing in his teens. “Boxing was a part of the curriculum when I went to high school,” he told Rolling Stone in 2009. “Then it was taken out of the school system, I think maybe in ’58. But it was always good for me because it was kind of an individualist thing. You didn’t need to be part of a team. And I like that.”

Even in his early days of dabbling in pugilism, it’s clear that boxing’s appeal to Dylan was always philosophical as well as physical. It didn’t just keep him physically active, it planted the seeds of American Individualism that factors so strongly into his later work. It spoke to his sense of social justice as well. And it became the occasional muse for his dayjob as one of the most influential singer songwriters of his time.

In 1963, shortly after featherweight champion boxer Davey Moore passed away as a result of injuries he sustained while trying to defend his belt against Sugar Ramos, Dylan penned the song “Who Killed Davey Moore?” a soul-searching take on the way in which everyone related to the boxing industry can put fighters at risk. A year later, he made a shout out to Cassius Clay in “I Shall Be Free No. 10.” Just over a decade later, in 1975, he co-wrote “The Hurricane” with Jacques Levy, a protest song the racial profiling that lead to the boxer Rubin “Hurricane” Carter’s unjust murder conviction (which wasn’t overturned until 1985).

Boxing also helps Dylan’s art by keeping his most important instrument—his body—finely tuned. It remains his primary and favorite source of exercise. He did dabble in kung fu for a brief period of time, studying with David Carradine, but it didn’t stick.

“Bob took some lessons,” the actor told Uncut in 2003 in an interview to promote Kill Bill 2. “He didn’t really stick with it, but he had some fun together. My master used to come out to my house and teach me every morning, and I thought Bob could profit from it, so went over to his place, and he and his kids took a few classes with us. It was pretty cool. Bob was funny, y’know—anybody who’s just beginning with kung-fu tends to be kinda funny, anyway, but he didn’t stick with it. But Bob’s an amateur boxer. He knows how to take care of himself. I know he seems like just a little wimpy guy.”

The singer did keep up with the sweet science, though, and has amassed perhaps the most star-studded collection of sparring partners of any professional, amateur, or hobbyist boxer that the world has ever seen.

In the middle period of his career, he trained pretty seriously with the middleweight Bruce “Mouse” Strauss, whom he called his “professional opponent.”

“Mouse could walk on his hands across a football field,” Dylan said in his 2009 Rolling Stone cover story. “He taught me the pugilistic rudiments back a while ago, maybe 20 or 30 years.” The pair got into a number of vigorous sparring sessions over the years.

Dylan had clearly followed Dempsey’s advice. He wasn’t afraid to hit hard. And he wasn’t, as Gina Gershon discovered, afraid to take a punch when he a bit younger, either. As she told Interview Magazine, the music icon somehow became one of her sparring partners when she was preparing for the 1996 cult classic Bound. And she decked him.

“One time he gave me a little jab in the face, and since I’d told him not to do that, of course I went insane and hit him really hard,” Gershon said. “He did go down, and I almost started to cry, thinking ‘Oh my God, I’m the jerk who broke Bob Dylan’s jaw.’”

But Dylan was fine. “He just said ‘Oh no, I need a good woman to kick my ass every now and then.’”

When he’s on the road—which he almost always is—he’ll occasionally stop in at various gyms along the way for a round or twelve. In 2008, he dropped by Nuevo Jordan Gym in Mexico City. Trainer Rodolfo “Guergo” Rodriguez didn’t recognize him when he first started sparring. “I said ‘These old guys are going to give each other heart attacks!’” Rodriguez told the Mexican newspaper El Universal. “Dylan boxed with all his friends and he did well; you can tell he’s practiced for a while because he landed some good shots and brought his own professional [equipment]. He knows what pugilism is and enjoys it.”

He’s also a recurring guest star at Lord’s Gym in Austin, the gym run by the carrot cake-baking boxing coach who once taught David Bowie. Lord will occasionally recruit older sparring partners for Dylan, asking them not to hit the star. Comedian and ad agency creative Daniel “Catfish” Rush recounts one such encounter:

“We step in the ring. He does not look at me. There is no conversation. The round bell goes off. He’s old and tiny but he makes his way to the center of the ring, throws a series of jabs that don’t reach. Neither does his right. He throws a hook. I take it on my arm. I dance around, slip left and right. Before I knew it, the round bell rang.

“Dylan goes back into his corner. Richard mouths some advice to him. The next round begins. Apparently the advice was to throw more hooks. They all miss or they land on my right arm. I’m just dancing around. Every once in a while, I get into a cat crouch and lean in so he can land shots. He hits me on the forehead with a straight right. Pretty good shot. Bell rings. Second round is over. ‘Thanks,’ Richard says. ‘That’s all, Catfish. Thanks.’”

He’s also been known to drop in to watch his favorite pros train. Last year, he visited Manny Pacquiao before his bout against Timothy Bradley. “It was like seeing one of the apostles,” Pacquiao spokesperson Fred Sternburgg told the press “It was like seeing one of the apostles.”

Dylan spends most of his boxing-related hours at his own gym now, though, a semi-secret fight club in California. Will Smith and Sean Penn have trained there. Magic Mike’s Joe Manganiello likes to brag about his training at the club.

Former lightweight champion Ray “Boom Boom” Mancini calls it the best gym he’s ever been in. “On the wall there are pictures of Joe Louis, Ali, Frazier, Muddy Waters, the Rolling Stones. The heavyweights of boxing and music,” he told Boxing.com in 2013.

Mancini also had a chance to get into the ring with Dylan. “First time I was over there we were sparring and just to keep him honest I would tap him with a left or a right. After a couple of rounds, Bob said ‘Hey Ray. Could you take it a little easy on the head? I have a few songs left in there.’ I was like oh no—are you kidding? Sorry. But he was only joking. He wasn’t real fast or strong but he had his own way of moving and could get it done.”

Which is exactly what he did when he welcomed filmmaker Quentin Tarantino to his gym and sparred with him. “He wanted to play hard... [and] he nailed me with a really good one [punch],” Starpulse reports him as saying. “We were sparring together, and he got one in there when I wasn’t paying attention. I think it was a right jab. I let my guard down for a minute, and he just thumped it in. It was a good punch.”

Dempsey would be proud.

Check out these related stories:

David Lee Roth Is The Best Martial Artist in Rock

David Bowie and the Boxing Coach Who Showed Him the Ropes

Don’t Hit Me in the Mouth, I Gotta Play Tonight: Miles Davis and Boxing