Lord Smith, the chairman of the Environment Agency, has refused to apologise for the flooding crisis during his first visit to the stricken Somerset Levels.

Smith ducked repeated invitations from reporters to say sorry and said he would not resign but insisted he was there to help work out a long-term solution to the area's problems.

Asked why his first stop was not one of the flooded villages but a high – and therefore dry – willows and wetlands centre, Smith said he did not want to get in the way of emergency workers.

Many locals said they were glad he had finally paid a visit. Others were irritated that he did not yet seem to have a solid plan about how to prevent flooding in the future.

Emerging from a meeting with people whose homes and businesses have been flooded, Smith said major dredging of the rivers – a key demand of residents – was now possible because David Cameron had made £10m available for the work. "That's what we need to get on and do as quickly as possible," he said.

"Beyond that we need to find longer-term solutions for the protection of the Somerset Levels and that means looking at things like land use, the way the upper catchment is used, stopping water coming into the levels, looking at the exit of the water into the Severn estuary. That is longer-term work and is even more important in the long term than the dredging."

Smith refused to apologise. "What I have been doing is explaining what the situation has been over recent years, the funding constraints, the Treasury rules we have had to abide by, the money that we did put on the table for dredging and the work that our staff at the Environment Agency have been doing day in and day out. We've had the biggest pumping operation we've ever had over recent days and I pay a lot of tribute to the staff who have been doing that."

Smith said he remained "very proud" of the work of his staff, and insisted that the top priority for authorities was "protecting lives", followed by protecting homes and businesses.

The peer said: "I have no intention of resigning because I'm very proud of the work the Environment Agency and its staff have been doing right round the country in the face of the most extreme weather."

But Jim Winkworth, a farmer and landlord of the King Alfred pub in the flood-hit village of Burrowbridge, who attended the meeting, said Smith had failed to provide answers to his questions.

Asked what he thought of Smith's refusal to apologise, he said: "Bloody mad. We thought that's the least he could do today and he's not apologising or admitting any liability. He hasn't come down here to apologise, which is what he should be here for.

"If you apologise it means you're admitting you got it wrong, I made a mistake, I'm sorry, I messed up but he's not fit to do that."

Winkworth, who is a member of the Flooding on the Levels Action Group (Flag), which is fighting to get the rivers on the Levels dredged, said Smith's successor – he is due to step down later this year – needed to listen to locals and "actually kick some arse, get some work done".

Winkworth accused Smith of "letting everyone down". He said: "He is the man in charge with the answers and he should be giving us the answers and sadly he is the man who is giving us the answers that we do not think are correct. He may think he is correct, but he is not. He is letting himself down, he is letting his organisation down and he is letting us down."

He said Smith had explained to him that the dredging that was promised a year ago had not happened because the £400,000 set aside was not enough. "So they sidelined the £400,000 and they were waiting for other agencies to give them more money to get started," Winkworth said.

"So my question was, why didn't they use the £400,000 to make a start? He said it wouldn't work and there was no point starting a job and not finishing it."

Liz Parris, 42, a company director who was evacuated from her home in Moorland with her husband and two dogs on Thursday morning, was waiting outside the meeting to hear what Smith had to say.

Parris, 42, who has a foot of water in her home, said: "From a personal level, people are very angry. Lord Smith is here now and I am not going to undermine talks with him. Let's see what he has to say, let's give him a chance.

"People are angry, people are very cross with him. I have to say the EA guys on the ground have been absolutely fantastic. The issues have been, in some cases, with higher up the organisation. The people who have farmed this area, lived in this area, managed this area for many, many years were calling before Christmas to turn on the pumps and drain water off the land.

"That was ignored. I hope Lord Smith is listening to what we have to say and something good will come of this visit."

Smith's visit on Friday – a week after the environment secretary, Owen Paterson, was heckled by residents – came as Royal Marines were helping evacuate about 140 properties in the village of Moorland.

Another night of heavy rain overwhelmed local flood defences and, despite advice from police, a handful of people had chosen to remain in their homes.

About 5,000 properties have been affected by flooding across the country, including 40 in Somerset, and flooding is expected to continue in the coming days, with severe weather alerts in place for Wales, south-east and south-west England as more storms are forecast.

While Smith was in Somerset, the transport secretary, Patrick McLoughlin, travelled to Devon to see the damage to the railway line in Dawlish, which will take about six weeks to repair.

The deputy prime minister, Nick Clegg, said he wanted answers from the Environment Agency, but dismissed criticism of Smith for failing to go to Somerset earlier, saying it was not right to "get under the feet" of the emergency services.

Speaking on his regular LBC radio phone-in, the Liberal Democrat leader said: "The Environment Agency have a very difficult job but also has to learn lessons from what has happened."

David Cameron, who chaired a meeting of the government's Cobra emergency committee on Thursday night, said everything that needed to be done would be done, and he would visit Somerset himself "when the time is right".

"I want to reassure everyone affected by these terrible storms that we will continue to keep up our urgent response and help get them back on the move," he said. "Progress is being made. We've announced extra money to go into flood defences. We've announced that we'll start dredging as soon as we can. We've got the army helping out the Environment Agency, including in Somerset and other parts of the country."