ESPN Colts reporter Mike Wells discusses Jim Irsay's decision to fire general manager Ryan Grigson, with putting together a better roster being the goal. Wells also breaks down Colts players excitement over the change. (2:12)

INDIANAPOLIS -- The Colts have fired general manager Ryan Grigson after five seasons, with owner Jim Irsay saying Saturday it was the right decision at the time for the team.

Grigson was under contract for three more years -- as is head coach Chuck Pagano.

But Irsay ultimately decided to part ways with only Grigson, while keeping Pagano in place for 2017.

"It was gut, intuitive, instinct from looking at where we were and where we are as a franchise," Irsay said. "I think that we needed to make a change, I felt. Intuitively you get the feeling when the timing's right where a change will help and whereas continuity is something that I really want and long to have as much as we can have it, but in this case I really felt the time was right to make a change. We needed some new direction just in the vision of our football program. That's from talking to a lot of people and giving it a lot of thought."

There also had been uncertainty about Pagano's future, but Irsay left no question there: "He's our coach for 2017."

The Colts went 8-8 this season, finished third in the AFC South and missed the playoffs for the second straight year.

Irsay said he has already compiled a list of candidates -- one that will not include Peyton Manning -- and plans to start an extensive search for a new GM. Vice president of football operations Jimmy Raye III is a candidate for the GM job and will head to the Senior Bowl in Mobile, Alabama, with Pagano next week.

Irsay has franchise quarterback Andrew Luck to dangle in front of potential candidates for the job.

"Have a list, and you know, it's not an impossibility somebody could be added to that list," Irsay said. "And, again, this is something that is a big decision. There's a lot of bright, young people out there. Obviously analytics and different aspects are affecting our game, how we go about being the best we can, and we worked hard to stay on the cutting edge. But, you know, that list could definitely change, though, and be added to, but we have a list that we are currently going and setting up people to come in and discuss the job with us."

Grigson, a first-time GM at the time of his hire, quickly made the Colts a winning team after replacing the fired Bill Polian in 2012. Grigson's first draft was highlighted by the selection of Luck with the No. 1 overall pick. Grigson also selected tight ends Dwayne Allen and Coby Fleener and Pro Bowl receiver T.Y. Hilton in that draft.

The Colts went 11-5 and reached the playoffs in 2012, and Grigson was named NFL Executive of the Year by Sporting News.

Indianapolis followed that up with two more 11-5 seasons that included back-to-back undefeated runs through the AFC South and an appearance in the AFC Championship Game. But the Colts have missed the playoffs in each of the past two seasons, including 2015, when they believed they were good enough to make a run at the Super Bowl following the offseason additions of running back Frank Gore, receiver Andre Johnson and linebacker Trent Cole.

None of the players Grigson selected in the 2013 draft are on the roster. His 2013 free-agent class -- highlighted by linebacker Erik Walden, offensive linemen Gosder Cherilus and Donald Thomas, cornerback Greg Toler and safety LaRon Landry -- was a failure for the most part. Walden is the lone holdover, and he's headed into free agency.

Former Colts GM Ryan Grigson has joined the Cleveland Browns as a top adviser. AP Photo/AJ Mast

Grigson didn't make a full commitment to improve the offensive line until the 2016 draft, when the Colts used half of their eight picks on linemen. That commitment came only after Luck missed nine games due to injury in 2015.

"I don't look at Ryan's time as a failure," Irsay said. "In this thing, I know only one team wins a Super Bowl each year and that sort of thing ... I'm just interested in us being able to bring that Lombardi Trophy back to us. So if it's a young guy who comes in and impresses without experience, it's still possible that someone could come in that hasn't been a general manager that's in his late 30s or whatever."

There was a disconnect between Grigson and Pagano, with their relationship so strained that they barely talked during the 2015 season.

In addition, several current and former players said it felt like Grigson had no desire to develop a relationship with them, and multiple sources said his perceived arrogance rubbed many inside the organization the wrong way.

But despite Grigson's flaws, Irsay gave him a contract extension in January 2016 because he felt like the general manager and Pagano could work out their differences.

"I untied 'em," Irsay said about Pagano and Grigson. "I can only tell you that, because it was in the best interest of the franchise. Chuck did not ask for Ryan to be fired, it wasn't 'me or him' or anything like that, neither did Ryan. My decision was based on what's best for the Horseshoe, what's best to make us the best football team on the field."

In the wake of the news, some Colts players tweeted comments that many interpreted to be supportive of the firing, including punter Pat McAfee, who sources said had been fined by Grigson over things he had tweeted in the past.