After more than three decades in Hollywood, Tom Selleck has dealt with his share of rumors. “They said I was gay for a while, which just wasn’t going to happen,” the Blue Bloods star says with a laugh, sitting inside a restored 1910 hunting lodge on his Ventura County, Calif., ranch. But there’s one persistent fabrication that has gotten to him: “Anywhere I go, somebody says, ‘Don’t you have a house in Boca Raton? Or New Jersey?’ or wherever,” he says with a baffled expression. “The truth is, I’ve only got the one place.”

But with a spread like this one, who needs a second (or third) home? On 65 acres of land, the property boasts a 1926 ranch house, a horse corral and a 20-acre working avocado farm; avocados are harvested in late spring, though Selleck admits, “It’s hard to make a living, let alone a profit” from his crops. Still, the ultraprivate ranch, says the star, “has been a real blessing.”

Get push notifications with news, features and more.

It’s also been the ideal escape from Hollywood. Once focused mainly on his career-as the star of the ’80s smash hit Magnum, P.I., “I worked 90 hours a week and did a movie every break”-Selleck, 67, took a big step back from the spotlight in 1988 when his wife, actress Jillie Mack, 54, gave birth to daughter Hannah. “I quit Magnum to have a family,” he says. “It took a long time to get off the train, but I try very hard to have balance, and this ranch has helped me do that.”

As part of that balance, he works only every other week taping Blue Bloods in New York City and shoots his popular annual Jesse Stone TV films (the latest one, Jesse Stone: Benefit of the Doubt, airs May 20 on CBS) in less than a month on location in Halifax, N.S. “My first priority,” Selleck says, “is time with my family.”

If there’s a secret to maintaining his nearly 25-year marriage, that may be it. “It’s important to nurture your marriage,” he says. “I think wanting time away from work has been good for that.” Selleck and Mack also attribute Hannah’s maturity to their country lifestyle. “We both thought,” says Jillie, “it was the best environment for her to grow up.”