(CNN) Democratic presidential candidate Bernie Sanders offered on Wednesday night as close to an apology as he could when pressed on why he -- as the former chairman of the Senate Veterans Affairs Committee -- waited so long to act on the burgeoning wait list scandal at the VA.

When pressed by CNN's Anderson Cooper on Sanders lack of action, the Vermont senator responded, "We should have done better."

The admission came during a CNN town hall forum in New Hampshire hosted by Cooper, who confronted Sanders with the candidate's own record.

Cooper pointed to 18 VA inspector general reports detailing problems plaguing the VA and asked Sanders why it took him so long to act.

"Well, a fair question. And I think you know the answer is that we have worked on many, many issues," Sanders said. "And your point is fair, that we should've acted sooner. We should've known what was going on in Phoenix, those long waiting lines and the lies that some administrators were telling us."

It is an issue that could come back to haunt the Vermont senator in his bid to win over the very military veterans and families he vowed to serve during his two year chairmanship on the Veterans Affairs Committee.

Veterans groups and others criticize Sanders for what they call a lack of oversight of the VA, and for at times coming to its defense in the midst of the scandal that rocked the agency in 2014.

Paul Rieckhoff, founder and CEO of the Iraq and Afghanistan Veterans of America, said Sanders largely ignored the appeals of organizations like his during a time when media and government reports exposed how veterans were waiting months for appointments and VA officials were covering up the delays.

"For far too long he was apologizing for the VA. He was refusing to acknowledge the severity. He was positioning it as a smaller issue than it was while veterans were dying waiting for care," Rieckhoff told CNN.

For more than a year, CNN reported on veterans' deaths and delays at VA facilities across the country, including detailed investigations in November 2013 and January 2014 examining deaths at two VA facilities in South Carolina and Georgia.

An internal VA audit later uncovered faulty scheduling practices and a "systemic lack of integrity" within some VA facilities.

Rieckhoff says Sanders had to be ignoring what was happening on the other side of the Capitol during this time, when House Committee on Veterans Affairs investigators were digging up records, swearing in whistleblowers and exposing the massive scandal.

During the time the House VA committee held 42 separate hearings related to oversight of the VA, the Senate VA committee -- chaired by Sanders -- held about seven hearings on these issues.

At the boiling point of the VA scandal in 2014, after CNN reported about secret wait lists at the Phoenix VA, the Senate committee held a hearing in which Sanders supported an independent investigation. Yet he also praised the VA for serving 200,000 veterans a day and cautioned about rushing to judgment.

"The point I want to make is that when you are dealing with 200,000 people, if you did better than any other health institution in the world, there would be thousands of people every single day who would say 'I don't like what I'm getting.' And we have to put that all of that in the context of the size of the VA," Sanders said at the hearing on May 15, 2014.

Sanders appeared on CNN the same day and again cautioned against a rush to judgment.

"Did the delays in care of these people on the secret waiting list actually cause these deaths? We don't know," Sanders said.

The following week, Republican senators on the committee sent Sanders a letter complaining about the lack of action.

"As we have previously stated, the committee has held a limited number of hearings on the issues faced by the department ... Our nation's veterans do not deserve to wait to receive their benefits and/or needed health care services," the senators wrote in a letter.

The scandal reached its climax when VA Secretary Eric Shinseki resigned May 30, 2014.

Sanders did later acknowledge the severity of problems at the Phoenix VA and others and worked with members of the House VA committee to pass the Veterans Access, Choice and Accountability Act. One of the law's provisions is for funding allowing veterans to obtain private care if waits are too long within the VA.

Despite criticisms, some veterans groups have supported Sanders.

Veterans of Foreign Wars presented Sanders with its "2015 Congressional Award" for his work on the Senate committee.

"He has been a commanding voice ... for better employment opportunities and improved access to mental health programs, as well as increased congressional oversight of the VA claims processing transformation," VFW National Commander John Stroud said of Sanders earlier this year.

Another group -- Veterans For Bernie Sanders 2016 -- has more than 10,000 likes on Facebook.

Sanders defended his overall record on Veterans Affairs last night, highlighted by passage of the VA reform bill signed into law last year.

"We went further in than any time in recent history in improving health care to the men and women of this country who put their lives on the line to defend us," Sanders said, referring to $15 billion given to the Department of Veterans Affairs to decrease wait times and reform the troubled agency.

Dan Caldwell, political director of the conservative group Concerned Veterans for America, remains unconvinced.

"Senator Sanders will likely continue to try and tout his time as Senate VA Chairman. But an honest look at his time as chairman will show that he failed in his responsibility to oversee the VA and was not a leader in the fight to reform and fix the VA," Caldwell said in a statement.