Shortly after Father Patrick Conroy, the House chaplain who was controversially ousted from his position by House Speaker Paul Ryan has rescinded his resignation letter, Speaker of the House Paul Ryan said he can stay.

"I have accepted Father Conroy's letter and decided that he will remain in his position as Chaplain of the House," Ryan said in a statement Thursday afternoon. "My original decision was made in what I believed to be the best interest of this institution. To be clear, that decision was based on my duty to ensure that the House has the kind of pastoral services that it deserves. It is my job as speaker to do what is best for this body, and I know that this body is not well served by a protracted fight over such an important post. I intend to sit down with Father Conroy early next week so that we can move forward for the good of the whole House."

The attempt to force out Conroy has sparked tensions on Capitol Hill for the last week. Conroy's impending departure was controversial, and came with little explanation last month. But in a letter dated Thursday, Conroy told Ryan he is formally rescinding his resignation. Conroy told reporters Thursday afternoon, before Ryan issued his statement, that the "story's not over."

Get Breaking News Delivered to Your Inbox

In his letter, Conroy claimed Ryan's chief of staff, Jonathan Burks, met with him on April 13 and asked for a letter of resignation. When Conroy asked if he was being fired for cause, Conroy says Burks "mentioned dismissively" that "maybe it's time that we had a chaplain that wasn't Catholic." Conroy claimed Burks also mentioned a prayer the chaplain offered in November during the tax reform debate that was viewed by some as critical of the GOP platform.

Conroy is a Roman Catholic priest from the Jesuit order. Ryan himself is Catholic.

"I have never been disciplined, nor reprimanded, nor have I ever heard a complaint about my ministry during my time as House chaplain," Conroy wrote. "It is my desire to continue to serve as House Chaplain in this 115th United States Congress to the end of my two-year term, and beyond, unless my services are officially terminated (however that is properly done) or I am not re-elected to the position by the membership of the House."

Burks disputed Conroy's account, saying, "I strongly disagree with Father Conroy's recollection of our conversation. I am disappointed by the misunderstanding, but wish him the best as he continues to serve the House."

Conroy's comments also seem to contradict Ryan's initial comments, when the speaker said Conroy's departure had nothing to do with politics.

"Father Conroy is a good man and I am grateful for his many years of service to the House," the Wisconsin Republican said in Milwaukee last month. "This is not about politics or prayers. It was about pastoral services. And a number of our members felt like the pastoral services were not being adequately served or offered."

Lawmakers, including some Republicans, had urged Ryan to keep Conroy on as the House chaplain.

House Democratic Caucus Chairman Joe Crowley, D-New York, is still continuing his call for a select committee to look into the matter.

"Father Pat has served the House honorably for more than seven years, and I'm glad that he will remain the House Chaplain," Crowley said in a statement. "Still, because there are conflicting reports and questions left unanswered, we need a full understanding of what happened. This is why I've called for a select committee to lead an inquiry into the events leading up to his abrupt dismissal. I hope Republicans will join Democrats to help us get the facts and ensure that something like this doesn't happen again."

—CBS News' Kathryn Watson contributed to this report.