Millions of people may be unable to vote in May's general election because the Cabinet Office refused to heed warnings from its own pilot programmes two years ago.

Electoral Registration Officers (EROs) at local authorities who have spoken to us on condition of anonymity say the Cabinet Office – the secretive Whitehall department responsible for, ironically, “efficiency” – failed to heed the warnings from the pilots, which showed major problems with data matching.

EROs have new responsibilities because of a change in the way that the electoral register is compiled. The pilots indicated that under the new system, EROs would need more resources to fulfil the new legal burden of Individual Voter Registration, which required them to verify what was previously taken on trust.

This is the first General Election to use “Individual Voter Registration” (IVR), which requires each voter to register to vote individually.

The change was made in the 2013 Electoral Registration and Administration Act. Previously, the “head of the household” was responsible for ensuring voters were registered to vote.

In October, the Electoral Commission reported that 5.5m voters were missing from the Electoral Register. But this shouldn’t be a surprise to the Cabinet Office, who knew of the problems two years ago.

This is going to be tricky. Full steam ahead!

Throughout 2011 and 2012, the Cabinet Office conducted a series of pilots into how the new-style electoral registration process would work. You can read the conclusions here (pdf). The process put the burden of corroboration – by data matching – on to local EROs. The report showed that the Cabinet Office not only knew that it would produce problems, but which demographic groups would fail the data-matching tests.

“Overall, this suggests that the currency/accuracy of the address data on either database is likely to cause records to fail to match, and as a result certain groups are less likely to be matched under-confirmation”, the report concluded [p15]. The report told the Cabinet Office where problems would occur, too:

Areas with either higher proportions of students (aged 16-74) [sic], higher proportions of people living in private rented accommodation and/or higher proportions of people living in communal establishments had relatively lower match rates.

The move to IVR raised other issues, MP Julie Elliot told us last year, that have not been addressed:

“Having the wrong NI number is not uncommon,” she said. “35 per cent of Muslim women do not have National Insurance numbers. In principle, I agreed with individual voter registration, but I am concerned how it works.”

EROs also confirmed that only expensive and labour-intensive canvassing would fill the gaping holes in the Electoral Register.

In August, the Cabinet Office’s new in-house IT team, the Government Digital Service, launched a web front end to the system. GDS wasn’t shy about boasting about it.

“The digital service allows anyone to register to vote online, from any device connected to the web. It’s incredibly quick and simple – it takes less than five minutes. All you need is your National Insurance number (or your passport, if you’re abroad). It’s so quick, you could be forgiven for thinking 'Is that it?' when you’ve finished your registration,” wrote GDS boss Mike Bracken CBE, adding:

This is how democracy works now. Voters are users too, and this service was designed to meet their needs.

Perhaps some Ministers even believed what their creme de la creme tech team was telling them. Yet what GDS was telling the people wasn’t actually true.