Ronda Rousey joined the Ultimate Fighting Championship less than two years ago and hasn't lost a match. The 27-year-old mixed-martial arts star carries a 10-0 record and quickly subdues female opponents with moves like a swift kick to the liver or an arm restraint that leaves rivals yelping in pain.

But over the next year, Ms. Rousey—called "Rowdy Ronda" for her style inside and outside the UFC cage—is taking on a new challenger: Hollywood.

It hasn't always been an even match. As Dana White, the president of the UFC, put it: "Our people have starred in a lot of bad movies." (Some forgettable examples: "The A-Team" and "Cyborg Soldier.")

Ms. Rousey's multichaptered career has taken her from teenage Olympian to mixed-martial arts fighter to trailblazer: She's the first woman allowed to fight in the previously male-only UFC, a fast-growing fighting league marked by no-holds-barred matches in octagonal pens where the mats get covered in sweat, saliva and blood. Now, a new cast of trainers—from acting coaches to Hollywood agents—is set on helping her punch through another glass ceiling and become a viable female action star.

Her first test comes next month with "The Expendables 3," where she'll join muscled action stars like Sylvester Stallone and Arnold Schwarzenegger in the sequel about a team of mercenaries. It's the kickoff in a flurry of roles that include appearances in the coming film adaptation of "Entourage"—the HBO show about a young actor navigating his own Hollywood stardom—and a part in "Fast & Furious 7," the 2015 installment of the car-chase franchise known for its diverse, big-tent casts.