INDIANAPOLIS -- Colts general manager Chris Ballard ended his Wednesday news conference with a drop-the-mic line: "The rivalry is back on."

That was Ballard's reaction to the Patriots and their offensive coordinator, Josh McDaniels, who reversed field and decided to remain in New England rather than become the next head coach of the Colts.

According to Ballard, McDaniels called him on Tuesday and said he had bad news.

"I just said, 'I just need a yes or no answer. Are you in or out?'" Ballard said Wednesday. "We went around for a minute and he said he's out, and I said, 'OK, we're gonna move forward. I wish you the best of luck.'

"There was no persuasion. ... We have work to do. I want someone that's 100 percent committed to partnering with us and getting that work done."

Colts GM Chris Ballard will renew his coaching search after Josh McDaniels back out of their deal. AP Photo/Michael Conroy

Ballard says he quickly moved on from the five-minute phone conversation. Wednesday he put an emphasis on reassuring people that the Colts will be able to find the right head coach to replace Chuck Pagano, who was fired in January after six seasons.

The Colts identified McDaniels as their primary candidate early in the coaching search and were waiting for the Patriots to finish their run to Super Bowl LII. Now Indianapolis will interview New Orleans Saints assistant head coach Dan Campbell on Thursday, a source tells ESPN's Adam Schefter, and Philadelphia Eagles offensive coordinator Frank Reich will interview Friday, a source tells ESPN's Dan Graziano. The Colts also requested permission to interview Buffalo Bills defensive coordinator Leslie Frazier, a source confirmed to ESPN.

"There was no persuasion. ... We have work to do. I want someone that's 100 percent committed to partnering with us and getting that work done." Colts GM Chris Ballard

"It doesn't always work out or go the way we mapped it out or planned it," Ballard said. "But it's the mentally tough, gritty people that overcome these things that are successful. ...

"We're gonna get there. Unquestionably, we're gonna get there. We have a list of candidates. [We] had them from the get-go. ... There's other guys we wanted to interview [early in the process], but because of the playoffs, we weren't able to do it. We'll move forward with them, and we will get the right leader for the Indianapolis Colts -- one that believes what we believe and wants to go where we want to go. I'm very confident in this."

The Colts were so confident that McDaniels would be their next coach that they announced they had agreed to a deal with him on Tuesday and had scheduled a news conference to introduce him on Wednesday afternoon.

McDaniels' agent, Bob LaMonte, was stunned Tuesday when McDaniels informed him that he would not accept the Colts job, a source told Graziano. When McDaniels told him he was staying in New England, LaMonte replied that McDaniels was "committing professional suicide," according to the source.

LaMonte, who is also Ballard's agent, has terminated his relationship with McDaniels, according to sources.

Ballard took responsibility for announcing the hire before McDaniels signed a contract.

When asked for his reaction to McDaniels' decision, Ballard said, "I'm competitive, and I want to win. ... Things work out for a reason. Just because you're the first choice doesn't make you the right choice."

Several candidates the Colts interviewed early in the process, including Matt Nagy and Mike Vrabel, have already landed other head-coaching jobs. And the NFL's annual scouting combine is less than three weeks away. Despite being so late in the process, Ballard emphasized that the Colts will not put a timetable on when they will hire a coach. He also pointed out he won't circle back to any of the coaches they previously interviewed.

"There's a lot of good coaches out there," Ballard said. "There's a lot of good assistant coaches out there. Everybody gets in panic mode and starts hiring. I don't believe in that. I believe you have to be patient, take your time."