“Who’s the toughest actor in Hollywood?” Frank Grillo asks, wrapping his hands in preparation for training.

Grillo’s posing the question to Terry Southerland, his trainer of more than 20 years. Lacing up Grillo’s gloves, Southerland doesn’t hesitate before responding, “You are.”

Standing in the middle of an underground boxing gym in New York City, Grillo is just one of the guys. He walks like a fighter, he talks like a fighter, and other than having the best hair in the gym—according to his trainers—he looks like a fighter. In this moment, there’s almost nothing that points to the fact that Grillo’s been a working actor for more than 20 years.

After staying under the radar for much of his career, Grillo started making a name for himself with his breakout performance in 2011’s Warrior. Gavin O’Connor’s film told the story of two MMA fighters (Tom Hardy and Joel Edgerton) who also happened to be brothers. Grillo took on the supporting role of Frank Campana, Edgerton’s trainer—and attacked every scene with the intensity he’s bringing to this workout. “[My character] wasn’t in the script. That role I had to create myself, and I had to scratch and claw for every screen second,” Grillo says.

Grillo’s role in Warrior directly led to a more prominent role opposite Liam Neeson in The Grey, which helped open the door to the biggest year of his career. In 2014, Grillo headlined The Purge: Anarchy, found his way into the Marvel universe as Brock Rumlow in Captain America: The Winter Soldier, and launched a new show, Kingdom, on DirecTV, which has been picked up for a second and third season.

Suddenly, at the age of 49, Grillo is being talked about as an actor—and not just an actor who can kick your ass in real life. After all, this is the guy who nearly ripped a phone book apart on Watch What Happens Live!, and the same guy who told Jimmy Kimmel, “I could kill you.” He wouldn’t—but he could.

The easiest way to set off Grillo? “To be an asshole,” he says. “I can always tell somebody who’s never been punched in the face because they think they can say whatever they want.”

Now, surrounded by professional fighters, Grillo’s warming up on the punching bag. Every punch exudes an air of experience, a feeling that Grillo’s been here a million times before. On his face is a look of determination so intense you’d think the punching bag hit him first.

Yet the moment Grillo stops hitting the bag, his face breaks into a smile. Within seconds, he’s saying hello to another friend who’s just entered the gym. “This is my sanctuary,” Grillo says. “I’m either with my kids when I’m home, which is mostly, [and] on my free time, I’m here. I don’t do anything else. It’s kind of sad,” he adds with a smirk.

Grillo, a husband and father of three boys, started wrestling around the age of 10, then picked up jujitsu, football, and lacrosse before he started boxing in his late teens. “Now,” he says, “I’m 107 and still doing it.”

His fighting strategy: “Hit ‘em first. Hit ‘em hard. Hit ‘em a lot.” Interestingly enough, it’s advice he rarely follows anymore. As both an adult and as an actor, Grillo’s not nearly as reckless as he used to be; if you want to see him fight, you’ll have to catch him at the gym. “There’s just a truth here that’s rarely found outside. You can’t lie when you get in there,” he says, gesturing to the ring. “It’s hit or be hit. It’s no bullshit.”

Southerland and Grillo move from the punching bag to the mitts, as trainer Jason Lee makes his way across the gym to watch. “Frank gets in there and spars,” Lee says, gesturing to the ring. “He’s one of the only people in his position to do it. I used to train a lot of Hollywood actors, and no one would do it. The actors, they don’t get it. Frank gets it.”

Image zoom Samantha Highfill

When Southerland playfully taps Grillo in the face with his mitt, the trash-talking begins. As sweat drips from his face, it’s easy to see why Hollywood tends to cast Grillo in the role of “fighter,” “trainer,” or “badass.” It’s also difficult to understand why he never pursued fighting professionally. “To me, if you’re not going to be great, don’t do it,” he says. “Don’t get punched in the face just to be okay. I just don’t have that ability. I have no delusions of grandeur.”

Instead, Grillo took his skills to Hollywood, a place full of people who’ve never been punched in the face. It’s there that Grillo has gained a reputation for doing all of his own stunts, leaving Chris Evans black and blue after that elevator fight scene in Captain America: The Winter Soldier. “There were a couple times in that elevator scene where [Evans] said, ‘Dude, you’re hitting me,'” he remembers. “I go, ‘We are in an elevator. My stunt guy’s not coming in here. They won’t be able to shoot your face. We gotta do this.’ And we just start punching each other. We did that scene for six days. We were literally covered in black and blues. That’s why I love him.”

As Evans said at the British premiere of the film, “If you don’t block Frank Grillo’s punch, you’re going to get mashed. He doesn’t know anything but 100 percent.”

“There’s no one in Hollywood I wouldn’t get in the ring with,” Grillo says. “There’s no one. There are guys who are like me who train hard. Chris Hemsworth is one of them. He’s a worthy adversary. But I’ll get in the ring with anybody. Because there’s a lot of phonies, I’ll tell you that much. That’s why I live in New York. I go to LA and I see these guys; there are a lot of guys who like to put their chest up and put their chin forward and I’m always like, ‘Hey buddy, keep your chin down.'”

This fighter mentality oozes out of Grillo with every word—and he attributes much of his recent success in Hollywood to it. It’s why he’s a fighter first and an actor second. “I fought my way through the business, and I continue to [fight] for jobs, for respect,” he says. “So I would say I am a natural born fighter/survivor/actor.”

After an hour at the gym and a quick shower, Grillo heads to a nearby bar to grab a beer. By this point, he’s looking a bit more like a movie star—though still one who could take you in a fight. His hair is coiffed, his features sharp—yet his facial hair serves as the perfect reminder that Grillo’s not one of Hollywood’s clean-cut pretty boys. Instead, Grillo’s one of those guys who just wouldn’t look right without a little stubble. Part of you wonders if there’s even a razor strong enough to remove his.

Growing up in New York, Grillo was the first member of his family to fight or act. After earning a business degree at NYU—because “conventional wisdom” told him to—he started acting in “shitty little plays” and doing a “little bit of stupid modeling” until he found his way on screen in the soap opera Guiding Light in 1997. Oddly enough, he says, being considered “handsome” by Hollywood standards was the biggest detriment to his career early on: “When you’re in certain roles—roles that I’m interested in—most of those guys look like Gene Hackman. They look like Roy Scheider. They don’t look like they could do f–king swimwear modeling. So for a while I had to capitalize on how somebody might’ve responded to how I looked, because it was the only thing I had going for me.

“Now that I’ve gotten older and I’ve become a man, it doesn’t matter anymore,” he continues. “Now I just look like blue-collar guy, and I couldn’t be more happy about that. Look at guys like Brad Pitt—he suffered for a long time. Now, he’s Brad Pitt, he’s a beautiful guy and I’m not comparing myself to him, but it was a struggle for him to be taken seriously. He’s a movie star. When you’re an actor, a lot of actors don’t want to be movie stars. We want to be actors. We want to be Gene Hackman.”

After pausing the conversation for a brief cheers, Grillo stops to take in his surroundings. He goes on about the new bar, which just opened a block from his gym. For a guy who bases the length of his workouts on “how much alcohol I drank the night before,” this is a big deal. But the pause is short-lived. Before Grillo can fully finish that thought, his mile-a-minute mind flashes back to the conversation at hand, quickly diving into where he fits in the Hollywood landscape.

“It’s no secret that in [Hollywood], there’s kind of a void for blue-collar men. We usually go to England to find someone—or Australia, even,” he notes. It’s an interesting idea: When it comes to “guys’ guys,” why does Hollywood often turn to the Liam Neesons and the Jason Stathams? Are there more American actors who can fill that void?

Grillo has a theory: It’s all American society’s fault. “I believe we’ve come to a point where we’ve emasculated men too much. We’re trying to suppress a little too much of what nature has given us and I see a lot of weak men. I really do and it bothers me. Because my thing is, if somebody comes up to you on the street and you’re with your wife and family, are you going to be able to take care of them? That’s what I want. I’ll go down fighting but my wife and kids will get away. And that’s a big thing for me. I know it’s a little cave man-like.”

But does this really matter for actors? So long as they’re good at acting tough, does this supposed issue affect them? Typically, no. But Grillo says modern men’s lack of physicality has had an impact on the acting world. For a while now, Grillo has been attached to an American remake of The Raid, Gareth Evan’s Indonesian martial arts action film. The original film is known for its hand-to-hand combat sequences; the remake intends to incorporate the same type of martial arts. But there’s a reason that the movie, which was originally set to shoot in September of this year, still doesn’t have a start date: Its makers can’t find actors who can physically handle that style of combat. “The deal is, if we can’t get the right physical people, we won’t make the movie,” Grillo says. “So it just goes to show you the void of real men/physical dudes in Hollywood.”

But that void is also partially responsible for Grillo’s own success. He’s currently thriving in the blue-collar gap. And with a potential return in Captain America: Civil War, as well as a rumored third Purgefilm—not to mention his role in the on-hold The Raid remake—some will be quick to call him an action hero. But Grillo himself prefers “anti-hero.”

“I like guys on the wrong side of conventional wisdom,” he says. “It’s usually a person who’s at a low point in their life, has been maybe wronged by society or by himself or something’s happened to his family. Throughout the course of the film, there’s a journey that’s made, and there’s a change that happens as a man.”

Of course, being a badass doesn’t hurt either—which is one of the key things that attracted Grillo to his character in Captain America: “He is not good and not bad, Crossbones. He’s just a guy who’s defending his ideology, and he’s a badass. He has no super powers. He’s a mercenary.”

Grillo lights up when talking about this role, sitting forward on his stool to explain why Crossbones could beat any superhero—The Hulk included. “Here’s the deal about Crossbones,” he says, suddenly turning into a young boy arguing over which Avenger is best. “The Hulk is just a big thug who comes at you. He comes out swinging. Crossbones is much smarter than that. Plus, he’s got weapons. The Hulk has no weapons. He’s an idiot. And he’s green. Crossbones doesn’t lose to people who are green. I’ll fight any of those guys. Any of them.”

At this point, the bar’s starting to get more crowded as Friday afternoon revelers trickle in. When a man with a backpack walks past Grillo and bumps him, Grillo doesn’t move. He doesn’t stop talking. He doesn’t stop having a good time. But ever so subtly, his eyes follow the offender all the way through the bar until he lands at a table in the back. With the judgment of a seasoned fighter, you get the feeling that Grillo always has a sense of what’s going on around him. But he says nothing.

In fact, the only time he appears to lose his cool comes about 20 minutes later, when he can’t remember the name of his favorite children’s book. After five minutes of frantically Googling and even calling his wife, Grillo remembers: “Where the Wild Things Are!” he shouts in the middle of the bar. “Wow. This is what happens when you’re 75 years old,” he adds, laughing. “Oh my God. Crack is whack.”

Talking about reading to his sons every night, and fatherhood in general, makes Grillo visibly emotional. “I don’t say this out of hubris: I’m not great at anything, but I’m a great father,” he says. “My goal in life is to keep them safe and secure and happy and make them successful. A lot of what I do is based on how I can take care of them. Do I want to be the fourth or fifth guy in a little movie that’s going to get a lot of attention in award season, or do I want to go and do a movie that’s going to make me a lot of money? Probably the latter.”

It’s a sentiment not many actors would express to a reporter—the idea that sometimes, money is more important than craft. But don’t mistake Grillo’s honesty for a lack of passion. When prompted, he can list off his dream projects without hesitation, beginning with a Deathwish remake and a really good gangster movie. Grillo, who met with Martin Scorsese for Wolf of Wall Street, would love nothing more than to team up with the Oscar-winning director. “If I could work with Marty Scorsese, you could shoot me. I’m done. I’m finished,” he says.

Other than that, Grillo’s Hollywood wish list includes working with Daniel Day-Lewis, Mark Rylance, and hopefully dipping his toe in the comedy pool—think a dark comedy along the lines of In Bruges. “I’m a funny guy!” Grillo says, almost as if to convince Hollywood from 2,000 miles away.

But when Grillo so much as suspects that the words “romantic comedy” are about to come up, it’s as if a security alarm goes off in his head. By the time he hears the phrase’s fourth syllable, he shuts the suggestion down. “You’ll never ever see me in a romantic comedy. Look at me,” he says, with a stare more intense than the one he gave the punching bag earlier. “Ever.”

The only other question Grillo answers with such a sense of finality is whether his sons—one of whom, Rio, is named for Grillo’s love of Brazilian jujitsu—will be fighters. “They’ll all know how to fight,” Grillo says. “But they’ll be poets before they’ll be fighters.”

Taking the last sip of his beer, Grillo checks his phone, realizing that Rio has a soccer game at 8 a.m. tomorrow—which means Grillo’s biggest opponent right now is the 17-degree weather that awaits him. That’s about as far ahead as he’s thinking. As a student of Eastern philosophy—something that helps counteract his occasional aggression—Grillo believes in living in the moment, whether that moment is head-butting some asshole (a move he does not recommend) or taking his wife of 18 years out for a date.

And in this moment, Grillo sees his career as a work in progress. “There’s still something I’m after,” he says. Surprisingly enough, it’s not mainstream fame. It’s not even clear if Grillo knows what he’s after. All he knows for certain is that he doesn’t want to be Brad Pitt.

“I don’t begrudge anybody,” he says. “I want everyone to succeed in whatever they want to succeed in and I’m not jealous of any other actor. I don’t want to be famous. I don’t want to be Brad Pitt. I don’t need to be on magazine covers. I just want to work with some cool people, have a good time, enjoy what I’m doing, make money, take care of my kids, always be challenged and go out of this thing saying, ‘I accomplished a little something.’