The full cost of introducing the government’s proposed mandatory data retention scheme is still not known, an inquiry has been told.

The parliamentary joint committee on intelligence and security is examining government legislation to force phone companies and internet service providers to store certain customer details such as phone numbers contacted, the time of calls, and the sender and recipient of emails.

Police and intelligence agencies already access metadata without a warrant, but argue laws are needed to standardise the information that telcos and ISPs store and ensure it is available for law enforcement purposes for at least two years.

Companies have warned that consumers could be forced to pay the costs of complying.

Anna Harmer, an acting first assistant secretary of the Attorney-General’s Department, told a hearing in Canberra on Wednesday that assessing the cost of the scheme was complex and ongoing.

“That work is continuing but also naturally depends on the ability of industry to provide relevant information,” she said.

Harmer said the financial impact on individual providers could differ widely because of “significant variability” in their existing business practices.

The shadow attorney general, Mark Dreyfus, sought clarity from officials about the government’s pledge to provide a “reasonable contribution” to companies’ capital costs.

Harmer said the government had decided to contribute to the initial costs of adjusting to the scheme such as reconfiguring networks “rather than bearing the entire costs of data retention”.

Katherine Jones, a deputy secretary of the Attorney-General’s Department, was unable to provide detail on the likely funding split, except to say the government’s contribution would be “reasonable from the perspective of industry [and] reasonable from the perspective of government and taxpayers”.

Officials however confirmed the metadata to be stored by ISPs could be sought by TV and movie companies for civil court cases against illegal downloaders.

Labor’s communications spokesman, Jason Clare, asked officials about the potential for owners of copyright-protected material to seek access to data for civil litigation against people found to be downloading movies, TV programs and music.

Harmer replied: “It is the case obviously that data that is already available and data that will become available in accordance with data retention is available and amenable to other lawful purposes including in the civil space, whether that is through subpoena or other orders for production.”

Such uses are not dealt with specifically in the proposed legislation, which focuses on the obligations on ISPs and telcos to store material to be specified by regulation. The government wants ISPs to retain the internet protocol (IP) address allocated to each user and when.

Police and intelligence agency chiefs told the committee hearing that metadata access was critical to criminal investigations, but that companies were increasingly storing less information or for shorter periods.

The Australian federal police chief, Andrew Colvin, said telecommunications data had been used in 92% of the AFP’s counter-terrorism investigations launched between July and September.

Such data had also been used in 100% of cyber crime cases, 87% of child protection investigations and 79% of serious organised crime matters, he said.

Colvin said the inability to access metadata would have “grave implications” for the ability to investigate, deter and disrupt potential terrorist acts.

He said different providers stored different information for different periods of time, underlining the need for standard retention requirements to be enshrined in law.

“When the AFP is dealing with serious threats to national security and other serious crime, we cannot afford to rely on luck to see if the provider the criminal has chosen to use has retained that data,” Colvin said.

The deputy director general of the Australian Security Intelligence Organisation (Asio), Kerri Hartland, said the agency regarded the proposed two-year storage requirement as a compromise because it would prefer a longer period.

“Asio has consistently made the case for such legislation over the past decade because of the value of communications data to security investigations,” she said. “It’s not a hypothetical discussion. Australia is a terrorist target, recent events have once again demonstrated that Australia is not immune from acts of terror.”

John Stanton, the chief executive officer of the Communications Alliance, told the committee there had been a “long and very co-operative relationship” between industry and law enforcement agencies. He said the industry had supported about 300,000 metadata access requests a year.

But Stanton said the legislation would be more accurately labelled “a data creation regime” because not all providers currently retained all material in the proposed data set.

He said it was still unclear whether the government would make a reasonable or substantial contribution to companies’ costs, and therefore the size of the financial shortfall to be paid “ultimately by their customers”.

“If parliament decides to bring the bill into law we think it is vital that the legislation be clear in its requirements, that it be practical in terms of the impositions it places on telecommunications service providers, that it avoids unintended consequences, that it is mindful of personal privacy and it is proportionate to the security challenges facing Australia,” Stanton said.