The Australian Bureau of Statistics recommended the government not put a new question on sexual orientation into the 2021 census, despite acknowledging that two federal departments need the data to deliver services and it would have “strong value across all levels of government”.

Documents produced to the Senate on Monday reveal that despite extensive consultation with LGBTI health groups about questions on sexual orientation and gender identity and an “identified need” for sexuality data, the ABS believed the potential new topics were too sensitive and risked public backlash.

Gender and sexuality questions were among eight potential new topics identified by consultation in 2018 and were set to be included in a census test in October before the ABS ditched the questions after assistant treasurer Michael Sukkar’s office intervened, according to evidence to Senate estimates by chief statistician David Kalisch.

The documents – tabled after a Greens order for production – reveal that in July Kalisch recommended to Sukkar that the only new topics for the census be on chronic health conditions and Australian Defence Force service, with old questions on household internet access and number of motor vehicles slated to be dropped.

The ABS nevertheless asked the government to consider gender and sexual orientation questions because “claims have been made about the need for information” on those topics which the government “is best placed to judge”.

The submission to Sukkar noted increased scrutiny of the census due to the 2016 census website outage, suggesting the government “may not want to progress any controversial topics at this time” as a result.

“There is an acknowledged risk of adverse public reaction to possible topics on gender identity and sexual orientation,” it said.

The submission noted that initial local testing and international experience suggests the adverse reaction is not “significant enough to impact on response rates or quality of data”.

In a more detailed topic brief on the sexuality question, the ABS said there was an “identified need from organisations such as the Department of Social Services, Department of Health, National LGBTI Health Alliance and a number of state and local government agencies for a reliable evidence base” to inform programs and budgets.

There was no “large scale” sources of data on sexuality, it said, and noted only those aged 15 years or older would be asked their sexuality.

“Submissions indicated that a new census topic on sexual orientation would have strong value across all levels of government and it is hoped that this would allow targeted support to be developed nationally and, potentially, in small areas.”

Nevertheless, the ABS noted “sensitivities and potential risks” with the sexual orientation question, including “lack of trust in government by respondents and privacy concerns which could reduce the accuracy of the data”.

On gender identity, the brief noted that stakeholders have argued the “significant lack of data available on LGBTI people … has led to inaccuracy in reporting and significant underestimates that have left LGBTI people relatively invisible in mental health and suicide prevention policies, strategies and programs”.

More data would help inform service provision in health, mental health and aged care, it said.

The brief noted the gender question “created confusion in some test respondents” in small scale tests, but did not detract from the response rate.

The ABS noted that both the sexuality and gender identity questions were “expected to generate public comment and debate” which would need to be “carefully navigated” by the government and ABS.

The ABS suggested the sensitivities and risks of the two new questions could be “evaluated further through a large-scale field test” before the questions were later dumped from the census test conducted in Wagga Wagga and south Brisbane.

The submission noted that the results of the October test were “unlikely to be available in time to help inform government decisions on census content”.

LGBTI Health Alliance executive director, Nicky Bath, said the revelation was “disappointing” because of the “great need” for gender and sexuality data which should be put above “the politics and individual viewpoints”.

Bath noted the UK and New Zealand are set to include the questions in their next censuses, but failure to include them in 2021 will see Australia wait until 2026.

The Greens’ LGBTI spokeswoman, Janet Rice, told Guardian Australia the documents failed to explain why the ABS “decided to do a complete about-face on their own decision to include gender and sexuality questions in the census field test”.

“This is what we know: the ABS told minister Sukkar the questions would be included in the field test. Then, without any obvious rhyme, reason or explanation, the ABS decided to remove those questions.”

Rice called on Sukkar to “come clean” about what contact his office made with the ABS, suggesting the decision to dump the questions from the test “looks suspiciously like [the product of] political pressure”.

“These questions are vital to ensuring that all Australians are counted. How is the government going to ever adequately address the needs of LGBTIQ+ communities if they refuse to count us in the census?”

The ABS recommended the question on sex be changed to offer a non-binary response option because the current binary question “cannot be answered accurately by people who are neither male nor female”.

The submission explains the ABS decided against potential new questions on journey to education and improved household and family measures, including shared care of children, because they were “too complex and expensive to collect”.

The ABS did not proceed with a new topic on Aboriginal and Torres Strait Islander cultural identity “due to Indigenous community concerns”, it said.

Smoking status was considered “of less value than other proposed topics for inclusion, such as chronic health conditions”.