More than three-quarters of Britain’s 59 butterfly species have declined over the last 40 years, with particularly dramatic declines for once common farmland species such as the Essex Skipper and small heath, according to the most authoritative annual survey of population trends.

But although common species continue to vanish from our countryside, the decline of some rarer species appears to have been arrested by last ditch conservation efforts.

“This is the final warning bell,” said Chris Packham, Butterfly Conservation vice-president, calling for urgent research to identify the causes for the disappearance of butterflies from ordinary farmland. “If butterflies are going down like this, what’s happening to our grasshoppers, our beetles, our solitary bees? If butterflies are in trouble, rest assured everything else is.”

While 76% of species are declining, prospects for a handful of the most endangered butterflies in Britain have at least brightened over the past decade, according to the study by Butterfly Conservation and the Centre for Ecology & Hydrology, with rare species responding to intensive conservation efforts.

During the last 10 years, the population of the threatened Duke of Burgundy has increased by 67% and the pearl-bordered fritillary has experienced a 45% rise in abundance as meadows and woodlands are specifically managed to help these species. Numbers of the UK’s most endangered butterfly, the high brown fritillary, are finally increasing at some of its remaining sites in Exmoor and south Wales, showing the success of targeted conservation efforts there.

Facebook Twitter Pinterest Duke of Burgundy fritillary is one species that has increased due to conservation efforts. Photograph: Peter Smith/Alamy

But The State of the UK’s Butterflies 2015 report cautions that such revivals still leave these vulnerable species far scarcer than they once were – the high brown fritillary has suffered a 96% decline in occurrence (meaning the sites at which it is present) since 1976, reflecting its disappearance from most of Britain. Other endangered butterflies, including the wood white (down 88% in abundance), white admiral (down 59% in abundance) and marsh fritillary have continued a relentless long-term decline.

The report reveals that the causes of the decline in rare “habitat specialist” butterflies, who are only found in specific places such as chalk grassland, are well understood, and usually linked to the destruction of flower-rich grassland or neglect of traditionally coppiced woodland.

But the reasons for the disappearance of once-common species from the wider countryside are less well understood by scientists. The wall – an orange and brown species that often basks on walls and stones – has declined in abundance by 87% since 1976 and by 77% in its occurrence. So it is now seen in both much lower numbers and in many fewer locations. Climate change and pesticides may be playing a more damaging role in their declines than previously thought.

Packham said it was imperative that scientists are funded to find out what is causing the loss of Britain’s insects.

“It points to the fact that there’s something significant that we’re missing,” he said. “I think personally it’s the over-use of a broad spectrum of insecticides and then neonicotinoids thrown into the mix since the 1990s. This is an urgent, clarion call to academics to get into gear and find out what is causing these declines.”

Common butterflies are generally faring better in Scotland than in England, mirroring trends reported in other species groups, such as moths. Scientists believe that climate change may be having a more beneficial impact in the north than in the south.

Facebook Twitter Pinterest The purple emperor has declined in abundance but increased in occurrence by 135%. Photograph: Alamy

Climate change is helping some species move northwards but increases in occurrence (or distribution) don’t necessarily translate into higher numbers: the silver-washed fritillary has flown into the Midlands and East Anglia in the last decade, with a 55% increase in occurrence, but its abundance has only increased by 6%. The purple emperor has declined in abundance but increased in occurrence by 135% – reflecting enthusiasts’ increased skill in spotting this elusive butterfly in previously unknown locations.

Climate change is providing some excitement for the thousands of volunteers who spot butterflies for Butterfly Conservation’s recording schemes, providing a uniquely detailed set of scientific records stretching back to 1976.

More migratory butterflies are reaching Britain, with the clouded yellow, red admiral and painted lady all increasing dramatically in abundance since the 1970s. In the last few years, rarer migrants traditionally only found in hotter climes, including the long-tailed blue, and European swallowtail, have also arrived in southern England.