The actor says he stopped receiving royalties after the Bill Cosby scandal led to reruns of The Cosby Show being pulled from syndication

Geoffrey Owens says the Bill Cosby scandal was just one of the factors that led him to search for work outside of his industry, including bagging groceries at Trader Joe’s.

Owens, 57, had been receiving royalties from The Cosby Show for years. But when Cosby, 81, was accused and later convicted of sexual assault, reruns were pulled from syndications, and the checks stopped coming.

Get push notifications with news, features and more.

“Yes, it impacted me financially,” Owens exclusively tells PEOPLE. “At the time that the show was pulled, that did make a difference in our income. That was one of the elements that led to my getting to the place where I said to myself, ‘I have to do something’ and I was thinking, ‘What can I do?’ and the answer ended up being Trader Joe’s, which is actually a wonderful situation for me in many ways. But I got to the point, I just had to do something to support myself and my family.”

After a customer took a photo of Owens, who played Elvin Tibideaux on The Cosby Show from 1985 to 1992, working at a Trader Joe’s in New Jersey, the actor says he was beyond “hurt.”

“I felt really humiliated,” Owens says of the photo. “From the time that I heard that the article might be done to the time it came out, I tried to envision the worse case scenario just to prepare myself and then it was just a little bit worse. If that was possible. They went out of their way to find the very worst picture of me, in the worst shirt and the worst posture. The words they used to describe me were so demeaning. It hurt.”

And while The Cosby Show star benefitted financially from the show’s success, he knew not to bank his entire career on the aftermath.

Image zoom

“I didn’t think I was set for life,” he says. “I didn’t think of it either way. I was single at the time, no family, and The Cosby Show paid me some fairly decent money for a single guy who never even expected to be on TV and was just happy to be doing theater. I felt like I had plenty of money, but set for life, no. But I was fine.”

He adds: “I always expected that one way or another I would work and make a living. Whether it was teaching, directing, acting, little job here and little job there, then I’d patch it together. I didn’t forsee working at a place like Trader Joe’s, as great as that place is. I didn’t foresee going out of the business for work, but I thought that within the business, I would patch together a modest living, at least enough to get by.”

Owens, who has since appeared on a number of shows including Divorce, It’s Always Sunny in Philadelphia, The Secret Life of the American Teenager and Built to Last, says steady roles are few and far between.

“The funny thing is, I never go too long without booking something, which is not a suprprise, the only issue is, the things that I book last one day or two days at the most,” he says. “I’ll book something for one day but then not work three or four months.”

But he couldn’t be more content with the life he’s leading now.