More than 100 teachers, writers and academics have said the government's early years education policies are damaging children's health and wellbeing.

The education specialists have written to the secretary of state, Michael Gove, to demand that children be allowed to learn through play instead of being prepared for formal lessons at such an early age.

Signed by 127 senior figures including Lord Layard, director of the wellbeing programme at the London School of Economics, and Sir Al Aynsley-Green, the former children's commissioner for England, the letter in the Telegraph says current research "does not support an early start to testing and quasi-formal teaching, but provides considerable evidence to challenge it".

It says: "Very few countries have a school starting age as young as four, as we do in England. Children who enter school at six or seven – after several years of high-quality nursery education – consistently achieve better educational results as well as higher levels of wellbeing."

Children must be receiving education by the age of five, and by age seven they are subject to three Rs assessment.

The letter, also signed by Dr David Whitebread, senior lecturer in the psychology of education at Cambridge University, Catherine Prisk, director of Play England, and the psychoanalyst and writer Susie Orbach, says current policy is heading in a direction contrary to global best practice.

"Though early childhood is recognised worldwide as a crucial stage in its own right, ministers in England persist in viewing it simply as a preparation for school. The term 'school readiness' is now dominating policy pronouncements" it says, adding that the government should not foist "the tests and targets which dominate primary education" upon four-year-olds.

The letter received a sharp rebuke from Gove's staff who said allowing children to play instead of learn was an "excuse for not teaching poor children how to add up".

"These people represent the powerful and badly misguided lobby who are responsible for the devaluation of exams and the culture of low expectations in state schools," a spokesman said.

"We need a system that aims to prepare pupils to solve hard problems in calculus or be a poet or engineer – a system freed from the grip of those who bleat bogus pop-psychology about 'self image', which is an excuse for not teaching poor children how to add up."

The letter was orchestrated by the Save Childhood Movement, which is launching a campaign called Too Much, Too Soon. The organisation will push for a series of reforms including a new "developmentally appropriate", play-based early years framework for nurseries and schools, covering children between the age of three and seven.

Wendy Ellyatt, the founding director of the movement, told the Telegraph: "Despite the fact that 90% of countries in the world prioritise social and emotional learning and start formal schooling at six or seven, in England we seem grimly determined to cling on to the erroneous belief that starting sooner means better results later."

Sir Michael Wilshaw, the head of Ofsted, said the best nurseries and primary schools had a "systematic, rigorous and consistent approach to assessment right from the very start".

David Cameron has previously championed wellbeing policies and has supported Layard publicly, saying the work of such academics was not "airy fairy".

• This article was amended on 16 September 2013. An earlier version said children must be enrolled in a school by the age of five. They must be receiving education, but do not have to be enrolled in a school.