Fewer than half of people with dementia living in care homes enjoy a good quality of life, a charity has warned, as it revealed that record numbers of people in care homes have the condition.

An Alzheimer's Society report states that 80% of people in residential care homes have either dementia or severe memory problems – a rise from previous estimates of 62%.

But the charity said that while excellent care exists, less than 50% of the 322,000 sufferers in care homes across the UK have a good quality of life.

It warned that society has such low expectations of care homes that people are "settling for average".

A spokesman said that 41% of more than 1,000 relatives and carers surveyed by the charity believed their loved ones enjoyed good quality of life, but 28% said it was poor.

A separate poll of 2,000 UK adults found that two-thirds do not feel the care sector is doing enough to tackle abuse in care homes, with many admitting they would be "scared" about moving into a care home in later life.

The charity called on ministers and the care sector to work together to strengthen existing minimum standards to boost quality of life for sufferers. They should also improve public understanding about the quality of care sufferers can be offered.

Jeremy Hughes, chief executive of the Alzheimer's Society, said: "When you walk into an excellent care home it's full of warmth, activities and interaction. But between these best examples and the worst, which often dominate headlines, there is a forgotten scandal of people with dementia who are failed and left living a life that can only be described as 'OK'.

"Society has such low expectations of care homes that people are settling for average. Throughout our lives we demand the best for ourselves and our children. Why do we expect less for our parents? We need government and care homes to work together to lift up expectations so people know they have the right to demand the best."

The health secretary, Jeremy Hunt, said: "No one should feel that they or their loved ones have to settle for poor quality care.

"Whilst many providers are delivering good quality care, this research clearly shows how more needs to be done to improve care across the board. There are no excuses for failing to keep people safe from abuse or not treating them with kindness, dignity and respect.

"This research shows the high prevalence in dementia is amongst people in residential care; it is essential that they are getting care that is right for them."