HOUSTON (CBS HOUSTON) – Brian Hoyer’s time in New England helped springboard his journey in the NFL where he readies as the starter for the Texans to take on the Patriots this Sunday.

“I remember when I was undrafted my agent was like ‘I think you should go to New England’ and I said ‘you’re crazy because they have Tom Brady.'”

Hoyer noted the team already had plenty of quarterbacks on the team, but he listened to his agent who sold him on the Patriots desire to have Hoyer and the ability to learn from Tom Brady and the Patriots coaching staff.

“I went in with that kind of mindset,” said Hoyer. “Just soak it all up. I don’t know if I’m going to make the team, if I’m going to be on the practice squad whatever, just soak it all up.”

Like a sponge.

“It ended up being probably the most valuable years of my NFL career because I learned from a lot of great people there. How to do it the right way. How to be a pro and have tried to apply that everywhere else I’ve gone.”

It wasn’t always easy though in his three years with the Patriots.

“It was tough,” said Hoyer when asked about playing for Patriots head coach Bill Belichick. “It was great to win, but it was tough. You always had to be on your toes. You always had to prepare like it was a game, even out at practice.”

Of course learning from Tom Brady was a tremendous opportunity for Hoyer.

Hoyer noted he spent almost eight hours a day with Brady and learned a lot about his routine, from when they film study starts in the morning to working hard on the practice field and in the weight room.

“The great thing about Tom was he kind of pulled me along with him,” he said. “For me, as a young guy, to see a player who is a great as he is to be that way and also holding me accountable.”

He remembered a story where during a break in the workday in New England the offensive linemen would play games in the locker room and he joined in with them. Brady approached him and asked him what to do on this play. Hoyer, a rookie, didn’t know.

He said Brady was adamant that he be ready should something happen to him Hoyer would be able to take over.

“Here’s a guy who’s won three Super Bowls, however many MVP’s, and he’s holding me accountable as his backup quarterback.”

His first stop the most important to date for Hoyer as he now gets ready to take down his old team.

“It was probably the most valuable time in my career because had I gone somewhere else I don’t know if I would be where I am today. Being able to be around (Brady), learn from (then offensive coordinator Bill O’Brien), learn from Coach Belichick. It was definitely a great opportunity.”

Where he is today, is staring down a playoff race, with the Patriots, and his old quarterback buddy, in the crosshairs.

Follow Stoots on Twitter – @Cody_Stoots