Indianapolis Colts owner Jim Irsay has been "very unhappy" with his team's performance for weeks now, and coaches in the organization have expressed fear this week that change is coming, league sources told ESPN.

There is uncertainty regarding whether just head coach Chuck Pagano's job is in jeopardy or whether general manager Ryan Grigson is also on the hot seat. But those in the Colts organization are bracing for change.

Irsay left the locker room in a golf cart after the Colts' 24-20 win over the Jacksonville Jaguars on Sunday without talking to the media.

Chuck Pagano has led the Colts to an 8-8 record this season. Kirby Lee/USA TODAY Sports

Pagano declined to address his job status when he was asked whether he planned to be the coach of the Colts again next season.

"Happy new year. Happy new year. It was a great win, really proud of this team, and can't wait to see my wife and my kids. I have my daughters here, my three granddaughters here. Baby Zoey, she is 3 or 4 years old, she doesn't give a damn about football right now. I am glad she came to the game," Pagano said.

"I remember when my girls were young, they didn't even want to go to the game, they asked if they could stay home with the sitter until they figured out what it was all about and these moments like that. This is why we do this -- for moments. We had a great moment there, and obviously we all celebrated what 98 [Robert Mathis] has done and meant. With that, happy new year to you guys."

Pagano might not be willing to talk about his job status, but Colts tight end Dwayne Allen spoke up by saying he thinks that the Colts should retain Pagano and Grigson.

"I saw a quote from Bill Belichick talking about having the head coach and your GM around. If you want to build championship teams, you need time in order to do that. Trading a coach out, trading the GM out, it's not how you build championship teams. Again, that's what Bill Belichick was saying. He's only arguably the greatest coach."

The Colts, who finish their season 8-8, have missed the playoffs for two straight seasons for the first time since 1997 and 1998.

Since playing in the AFC Championship Game in the 2014 season, the team has been slowed by injuries to franchise quarterback Andrew Luck and feuding between Pagano and Grigson over the building of the roster.

Although it seemed at the end of last year that Pagano or Grigson would be fired, Irsay surprisingly got both to agree to work together heading into this season and signed them both to deals that bound the two together for the next four years.

Pagano is 48-31 in five seasons with the Colts. In 2012, his first season as head coach, Pagano was diagnosed with leukemia. He missed Weeks 5 through 16, returning in time for the final week of the season.