Washington (CNN) Veterans Affairs Secretary David Shulkin says he has no plans to resign -- and that during a roughly half-hour conversation with White House chief of staff John Kelly on Thursday, he was not asked to resign and did not offer his resignation.

Shulkin told CNN on Friday that he was focused on making sure the agency was on track with its mission to get the VA working better for veterans.

"I'm not going to let the politics of what's going on distract me from doing the job that I came to Washington to do," Shulkin, who served as undersecretary of health for the VA during the Obama administration before being tapped to lead the department, told CNN.

A White House official familiar with the situation confirmed reports that President Donald Trump was upset with Shulkin and that the White House will not be defending him amid accusations that he abused his position for personal travel. Shulkin's meeting with Kelly was previously reported by The Washington Post

An investigation by VA Inspector General Michael Missal found "serious derelictions" by Shulkin and members of his staff during a July 2017 trip to England and Denmark, including that Shulkin's chief of staff altered an email and made false statements that led the department to pay more than $4,000 for Shulkin's wife, Merle Bari, to travel to Europe with her husband.

During the conversation with Kelly, Shulkin said, he shared an update on his plans to comply with all recommendations that the VA's inspector general had made, and they also discussed "how important it is to get everybody aligned around the President's agenda."

The same White House official said that some White House staffers felt that Shulkin had not given them a realistic view of the blistering report released by the VA's inspector general earlier in the week.

Multiple news outlets this week, including CNN, have reported on internal turmoil at the VA, including efforts by some staffers to undermine Shulkin's leadership. The New York Times reported on Thursday that some Trump administration officials were trying to replace him.

When asked about the internal maneuverings at the VA, Shulkin said there would be "no tolerance" for people who distracted from the VA's mission. "I'm not going to spend a lot of time on this. We're going to move forward," he added.

Shulkin said his goal was to "get everybody back on track, focused on the business of what we have to do to get the VA working for veterans." But he added that he was concerned that "not everybody is working together in a way that an effective organization does."

"As the leader of this organization, my job is to make sure that everybody understands that we're there to serve veterans and we have to be working in alignment if we're going to be successful at this task," he said.

Asked if he was concerned he was being pushed out, he said that he was "focused on the job, not with internal politics."

In the interview with CNN, Shulkin also weighed in on the decision of his former chief of staff, Vivieca Wright Simpson, to leave the VA after more than 30 years.

"This was a personal decision," Shulkin said, adding that Wright Simpson called him on Friday morning to inform him. "She just didn't feel that it was the right thing for her and her family to continue in that type of environment."

On Friday, the VA announced that Peter O'Rourke would replace Wright Simpson as chief of staff, effective immediately. O'Rourke currently serves as executive director for VA's Office of Accountability and Whistleblower Protection. O'Rourke's job will be "ensuring that the department works closely with the White House going forward," according to a statement from VA Press Secretary Curt Cashour.

O'Rourke is a Navy and Air Force veteran and previously worked for Trump's presidential campaign, according to his LinkedIn profile.