Government plans to throw money at automation and AI to develop public services risk "magnifying" problems around data quality residing in its own legacy systems, the National Audit Office has warned.

In its report today, Challenges of Using Data Across Government (PDF), the spending watchdog noted the government uses data to make millions of decisions every day, from checking border crossings to paying benefits and pensions.

Approximately three billion transactions take place each year, with the British government receiving or paying out at least £900bn through these interactions, it said.

However, it said the quality and availability of data in informing decisions remains a serious issue. In March, the Public Accounts Committee found the Home Office is making life-changing decisions using "incorrect data from systems that are not fit for purpose" and has not fixed the "appalling defects" identified during the Windrush scandal.

The report noted UK.gov is exploring the use of new technology such as robotic process automation and "artificial intelligence" in developing public services.

"However, layering new technology on top of existing data carries a significant risk of magnifying rather than overcoming the problems associated with data quality, for example if calculations cannot be adequately tested," it said.

Most of the government's data resides in its legacy systems, it noted.

"Departments have historically developed IT to support specific policy objectives. Even within the same department, data cannot be extracted or shared easily."

Some government departments have not always prioritised replacing older technology, it said, adding that "until they do so there will be ongoing costs and inefficiencies in decision-making."

The NAO previously found the Department for Work and Pensions had to use manual intervention when using data matching tools to help detect incorrect payments of Carer's Allowance.

"The newer systems provide more timely and accurate data matches, but the underlying data does not provide all the information that DWP needs, such as length of employment or allowable expenses. Officials looking at these matches therefore need to validate the carer’s circumstances manually and a shortage of staff to do this led to significant backlogs of cases."

It said often departments will produce a new customer interface, for example rolling out the online application for the state pension, without fundamentally reconsidering what data their department needs and whether their current data remains fit for purpose.

"In our report, Digital transformation in government, we reported that 17 of the government's 'Exemplar' programmes largely redesigned the user interface online and linked it to a pre-existing system."

Gareth Davies, head of the NAO, said in the issue is systemic across departments. "Government has lacked clear and sustained strategic leadership on data, and individual departments have not made enough effort to manage and improve the data they hold." ®