Journalist [archival]: Tonight with just six days to go, claims the census is in chaos.

Nick Xenophon [archival]: Right now we are looking at a complete shemozzle, a debacle, unless the government gets its act together.

Woman [archival]: I don't want to give my name, I don't think they need it, and who knows who can hack their data.

Journalist [archival]: One week after the census debacle there are reports that some Australians are still trying and failing to use the census online. But there is fresh scrutiny on the IT giant IBM. It has been blamed for failing to take adequate pre-emptive steps to deal with entirely predictable denial of service attacks.

Annabelle Quince: We all know the disaster the 2016 census has been, which is unusual because Australians have traditionally been willing to take part in the national census. But this time it was different; the privacy concerns, and then on census night the crashing or hacking of the online website.

Hello, I'm Annabelle Quince and this is Rear Vision on RN and via your ABC Radio app.

Rear Vision isn't into conspiracy theories but when a listener reminded us about the historical misuse of census data we thought it was worth a program, especially as the Australian Bureau of Statistics have given us an unofficial deadline of September 23, that's this Friday, to complete our census form.

While we are going to focus on the darker side of the census, let's just remind ourselves why a census is important. Margo Anderson is a Professor of History at the University of Wisconsin.

Margo Anderson: Censuses are formal time-defined population counts of a national state, and most countries do them now and most have for a good chunk of the 20th century. The function of the census is variable for different countries. In the US for example it's used to allocate seats in our legislatures and in particular our House of Representatives and in our electoral college. It's used as the baseline measure almost in all countries for economic statistics. So many of us are familiar with things like the labour force participation rate or the unemployment rate and so forth. And you need…for those rates you need denominators. And the census often serves as the baseline information for that. It's used for measuring growth and change and migration around the country. And often there are kind of cultural reasons that people, they mark the progress of their societies, if you will, by periodic census.

Annabelle Quince: And I'm wondering, is it the same reasons they collected data back 200, 300, 400 years ago or even further as we do today, or has that changed?

Margo Anderson: Well, the modern census is built on a notion of democracy. In other words, that people exist and have a right to exist and be counted and be identified. Prior to the development of Republican and Democratic governments, basically before the 18th century, censuses were taken often by kings or emperors or invaders to measure the military capacity or the tax capacity of their societies. And those were much less democratic notions. In other words, you had a king who basically said 'how can I extract revenue from my people, and to do that I need to know who my people are and where they are'. That function has by and large gone by the wayside. In fact it's flipped in most modern states where tax revenue is often allocated on a per capita count back to local areas. So I would argue that in the late 18th, early 19th century there was a fundamental shift in the notion of what a census was for, shifting it to the ruler looking at his ruled, to the people essentially standing up and expressing themselves. That has sort of been the dominant thrust ever since.

Annabelle Quince: But once a citizen stands up and is counted, even in a democracy, how do we stop that data from being misused?

Journalist [archival]: On 11 November 1918 an armistice had been signed. In Britain and France the survivors celebrated victory, the return of peace and the end of bloodshed. They left behind the nightmare lands of death and destruction created by four years of war.

Margo Anderson: In the United States during World War I the census was used to identify draft-dodgers. The selective service system, in other words the administration that was supposed to handle the draft, went to the census bureau and said we think people are lying about their ages. In other words, they were either saying they were too young or too old to be eligible for the draft. So give us the information from the 1910 census to tell us how old they are so we can draft.

Annabelle Quince: And were there any questions asked within the census organisation to say, look, we shouldn't be giving you this?

Margo Anderson: There was a big debate about it. And as often happens in wartime, lots of civil protections break down. And the use of the census during World War I was a fairly modest one, but it did happen. The census officials were quite uncomfortable with it, and essentially they basically strengthened the protections against that kind of intrusion after the First World War.

There were other more benign versions of this. During the 1920s in the US the Justice Department and the Labour Department went and asked for the ages of children in the 1920 census because they wanted to prosecute employers who were violating child labour laws. So there was an effort to protect the children by prosecuting the employer. And again, it was a similar kind of question; do you give out that information to facilitate that prosecution?

Annabelle Quince: How census data was used in Germany after 1933 is a reminder of why data protection is so important.

Journalist [archival]: Nazi propaganda was a reflection of Hitler's obsession; fanatical anti-Semitism. Hitler blamed Germany's 500,000 Jews for all of the country's problems. There was little reason for Germans to hate the Jews. They were a tiny minority, less than 1% of the population, and they had been patriotic and loyal. Tens of thousands had served and many died for the fatherland in the First World War. By the 1930s Jews were highly assimilated. Many had married Germans of other religions.

Edwin Black: What was a Jew in the mindset of the Nazis? Was a Jew a person who went to a synagogue and prayed on Saturdays? And the answer is no. A Jew was a racial Jew. Rassenblut was the eugenical definition of a Jew. The Nazis adopted the definition of the Carnegie Institution and the Rockefeller Foundation, that a Jew was not a Jew because of how he prayed and where he prayed, a Jew was a Jew by virtue of bloodline going back generations.

Annabelle Quince: Edwin Black is the author of IBM and the Holocaust.

Edwin Black: The Nazis wanted to know who were the real Jews. Was it just the guys with the black hats and the curls on their hair? Was it just the guys who looked like everybody else in business suits or went to synagogues? Or was it actually converted Jews, or people who didn't know they were Jews, worshipping in Lutheran churches and Catholic churches or no churches at all?

So what they did was IBM, which bills itself as the solutions company, came to Nazi Germany and said we have the solution. They came up with the racial census. So first of all IBM in and of itself, by itself, hired thousands and thousands of census takers, they went door to door, they did this for the government. All the information was all brought into one warehouse in Berlin, centralised. Day and night these paper forms were punched into special IBM coding machines. And then in one column they would have your mother tongue, whether it was Polish or German or Ukrainian.

In a second column they would have your religion, whether it was Jewish or Lutheran or Islam or a Catholic. In another column they would have your nationality; were you from Germany, were you from Poland, were you from France? In yet another column they would have your profession; were you a banker, were you a professor, were you a doctor? And then in a final column they would have your city, whether you were in Berlin or whether you were in Munich. And then at the rate of 24,000 cards per hour [snaps fingers], like that, suddenly they knew the identification and location of all the Jews in Berlin who were doctors. That was IBM.

Journalist [archival]: But the Nazis moved slowly against the Jews. The first official action came in April 1933 when propaganda minister Joseph Goebbels ordered a nationwide boycott of Jewish businesses. A week later, the Nazis expelled all Jews from the civil service and universities.

Edwin Black: There were six phases of the Holocaust, the first one being the identification of the Jews. That is what you are asking me in this census. They were then able to exclude the Jews, and that's the second phase, exclusion from society, by cross-tabulating census-identified Jews against the university rolls, against the labour union rolls, the medical societies. A third was confiscation. All the banks and financial institutions in Germany were running on IBM punch cards, so as soon as they could identify a Jew they would grab part or eventually all of their assets.

Journalist [archival]: In September 1935, Hitler personally introduced the Nuremberg laws, which deprived German Jews of citizenship and prohibited marriage or sexual relations between Jews and other Germans. In 1936 they imposed a special income tax on Jews. 1937, the Nazis began to pressure Jews to sell their businesses and urge German businesses to fire their Jewish employees.

Edwin Black: Fourth was ghettoisation, and ghettoisation means that the identified Jews who had been pauperised were now moved across town into these ghettos, 8 to a family, everyone knew exactly where they were going, it was all done on one day, it was organised, and this is the miracle of IBM's information technology. From there we go to step five which is deportation. All the trains in Europe were running on IBM punch cards. And so it was possible for a Jew to step out of his ghetto residence and travel for one to two days and get to a death camp, and 40 minutes later he is smoke. 40 minutes later he is gassed. No waiting. This is traffic management.

Journalist [archival]: SS Brigadier General Stroop in charge of the Warsaw ghetto in 1943 had learned his Nazi lessons well. In a secret report he said, 'The Reichsfuhrer SS ordered on 23 April 1943 the cleaning out of the ghetto with utter ruthlessness. I therefore decided to destroy and burn down the entire ghetto.' Little by little the Nazis were reaching what they called the final solution, the total extermination of the Jews of Europe.

Margo Anderson: What the Nazis did is that they took the information, the population information from the countries that they invaded, and used it to identify people to be rounded up and put in labour camps and ultimately exterminated. So in Norway, in the Netherlands, in France, once the Nazis took over the governments, they got into the data systems and used them essentially to control the population.

Annabelle Quince: And was there debates within the census organisations across Europe about what the Nazis were trying to do?

Margo Anderson: Yes, there were debates and there was resistance. What were used at the time was often what was called dot maps, in other words population density maps, so the Nazis were interested in, for example, identifying a Jewish neighbourhood in Amsterdam. So the census would have numbers down to fairly small areas of where Jewish families were on a street and so forth. And it was that kind of information, if given over to the military, could be used to facilitate roundups and so forth.

The census plays a minor role in what the Nazis did in the sense that they had many, many other means to identify Jews and undesirable people. But if you could get into the census figures, it sort of helped them.

Edwin Black: Most of the Jews who died did not die in Germany. They were exterminated in Poland, in Romania, in Eastern Europe, in Holland which had a nearly 75% death rate. The real mass murders of high numbers were in Poland. Would there always have been a Holocaust without IBM technology? Of course there would. But the Holocaust that we know, the Holocaust of the big numbers, the Holocaust that was organised, that became industrialised, mechanised, this was the Hollerith punch-card Holocaust, this was the IBM Holocaust, and there is no way to know if the high numbers of 99% here and 90% there, and 75% there could have possibly been achieved just with rifles and just with bullets. And so it was punch-cards, it was organisation, it was information technology.

Annabelle Quince: This is Rear Vision on RN and via your ABC Radio app. I'm Annabelle Quince, and we're looking at the historical misuse of census data.

Journalist [archival]: When the Japanese attacked Pearl Harbor, our West Coast became a potential combat zone. Living in that zone were more than 100,000 persons of Japanese ancestry. Two-thirds of them American citizens. One-third aliens. We knew that some among them were potentially dangerous, but no one knew what would happen among this concentrated population if Japanese forces should try to invade our shores. Military authorities therefore determined that all of them, citizens and aliens alike, would have to move.

Margo Anderson: The census was taken in 1940, and the census had a question on both national origin and on race, and 'Japanese' could be an answer to either one. So you could be Japanese by race, or you could be Japanese national, in other words somebody who immigrated and never became a US citizen.

After Pearl Harbor in December '41 there was great fear that there would be an invasion on the West Coast. And the military authorities in California requested what was called small area data from the 1940s census so that they could identify the Japanese-American population.

There was a racialist and racist reaction to the relatively small Japanese community on the West Coast that led to the congressional delegations and the governors and the media, the press, to call for the ousting of all Japanese from the West Coast, and their physical removal out of military areas.

And in February '42 Franklin Roosevelt in an executive order authorised that removal, and from late February, early March of 1942, so July '42 the Japanese-American population on the west coast of the US was rounded up and sent to concentration camps for the duration of the war.

The census, what they call small area data, was used to identify again the neighbourhoods and the city blocks and the communities where the Japanese-Americans lived.

Journalist [archival]: They gathered in their own churches and schools, and the Japanese themselves cheerfully handled the enormous paperwork involved in the migration. Government agencies helped in a hundred ways. They helped the evacuees find tenants for their farms, they helped businessmen lease, sell or store their property. This aid was…

Annabelle Quince: Was there much debate within the Census Bureau, within the United States, about using that material or that data in that way?

Margo Anderson: Yes, again there was, but Congress also debated it and Congress passed a law in February/March of 1942 eliminating the protection of statistical confidentiality on the census. And as a result it was, again, the consensus of the highest levels of American government that this should be the policy of the government. And of course it went to our Supreme Court and they also authorised it.

Years later the government apologised and paid reparations to the survivors and the people who had been incarcerated during the war, but that was not the sensibility during the war itself. Statistical confidentiality seems like a luxury that a country under attack cannot afford, and that was true during World War I and during World War II. The interesting thing in the United States is that it did not reappear either during Vietnam or Korea or the Gulf wars or the war on terror. So we've gone through a whole series of other wars since then where statistical confidentiality has not been challenged, or, if it has been challenged, the protection has held during those challenges.

Annabelle Quince: After the events of 9/11 there was an attempt to access census data in America, as Marc Rotenberg from the Electronic Privacy Information Center explains.

Marc Rotenberg: After 9/11 when the US had been attacked and the Department of Homeland Security went to the census bureau and said to the census bureau please give us information, in fact the data of the location of Muslim Americans living in the United States. And again you could sense very similar to what happened during the Second World War, the beginning of an effort to identify Muslim Americans. I don't think there was ever internment discussed, but at least some type of special measures. And we learned of this through a Freedom of Information Act that we had pursued with the Department of Homeland Security at the time.

The good news is that when the program was revealed, there was a genuine effort within the census department to improve its confidentiality procedures and to make clear that the information being collected by that one agency should not be made available to other federal agencies, even in those kinds of circumstances.

Margo Anderson: Interestingly enough that was close enough to the…there were still many, many survivors and supporters of the Japanese-American community who stood up almost immediately and said no, no, no, we're not doing this again. So there was a debate in the early 2000s about national security and identifying terrorists and the role of the census. There was not the kind of wholesale quite public abolition of confidentiality that there had been during World War II. The New York Times articles in early 1942 reported on what Congress was doing in abolishing confidentiality. That did not happen after 9/11.

Simon Lauder [archival]: What does the Australian Bureau of Statistics know about you? Every five years the national census creates a comprehensive and unique set of data about each Australian. The bureau keeps that data, but it can't link it to the answers you gave in previous censuses because it doesn't keep your name and address. That's all about to change, with this year's census. From now on the Bureau of Statistics will retain names and addresses. That means at each census, data can be linked to individuals and your profile of census data will become more comprehensive every five years and it can be linked to other government data.

Duncan Young [archival]: On order to take full use of that data, we need to be able to bring census data together with other datasets.

Nigel Waters: There are two main changes which have coincided. The first one was the change by the Bureau in deciding to keep the names and then generate from that what they call a statistical linkage key which is basically a unique identifier for every individual.

Annabelle Quince: Nigel Waters is a privacy consultant.

Nigel Waters: The second change is the decision to go online to facilitate the completion of the census online.

Annabelle Quince: That question of linkage, I thought the Bureau was saying that they would destroy the information after four years, that they just wanted to keep it for an extra period of time.

Nigel Waters: Yes, that's very misleading and disingenuous of them because after some pressure initially they were going to just keep names and addresses indefinitely. Around March this year, partly in response to a lot of concerns that were raised about the privacy implications, they announced that they would delete or remove the names and addresses and get rid of them after about four years, which is just short of the next census. What they didn't tell people is that in the meantime they will generate a statistical linkage key which is a unique identifier, so that even when they do remove and delete the names and addresses they will still know which census findings and results belong to which individual.

Annabelle Quince: Although if you look back say at the last 100 years or so of the ABS, have there ever been incidents where they haven't protected the data or they have misused the data?

Nigel Waters: Not that we are aware of, but that's largely because they weren't in a position to. The whole basis of the contract that the Bureau had with the public around the census was we'd like you to give us fairly sensitive detailed information about ethnicity, race, religion, family relationships et cetera, and the deal is that we won't keep that in a form where it can be linked to you as an individual. After the 18-month period that link was broken. So they were never in a position to have the temptation or abuse or the risk of abuse by hacking or any other sort of external threat.

Marc Rotenberg: I think we have two powerful trends colliding. I think the public awareness of concerns about the misuse of personal data is increasing. The other important trend of course is the increasing use, the commercialisation, the commodification of personal data by private companies, and they look at the data that is being collected by government agencies around the world and they see commercial opportunity, maybe it's advertising, maybe it's employment determinations or credit determinations, and they turn to the government agencies and through various means try to get access to the data for their commercial product. And that of course undermines the integrity of the census.

Annabelle Quince: Can you imagine that there could be incidents again that really test the statistics agencies?

Margo Anderson: Frankly no. I actually think that precisely because a lot of the misuse of the data has been explicated historically and that there is historical memory of doing things that you don't want to do again. There are other ways to abuse people's civil rights than making use of the census. It's a clumsy instrument. Up until the development of social media it was one of the major mechanisms, but now there is many, many other ways to do this. There is so much information coming through that people voluntarily submit to private organisations. There are new issues there about whether Google, Facebook, Twitter, et cetera have the same sensibility that the government does.

Edwin Black: The data is not safer, the data is faster. The data is not safer, the data is vaster. We now need to ask ourselves what are the controls we will use to preserve. After all, look at the regimes that we have in our world today. We have a Chinese regime, we have growing Islamic State regimes which takes control of census, tax and passport bureaus, we have Russia, we have repressive regimes in North Korea, we have repressive regimes throughout the Arab world, we have awful tendencies amongst our own democracies, and we use our census data here in a very sacrosanct fashion until it becomes non-sacrosanct. It's the emergencies, it's the anomalies, it's the special events, it's the nightmares, it's the horror stories that present themselves, that create the challenge to keeping this data safe and sound. And since neither you nor I can predict what the next challenge is, what the next September 11 is, what the next terrorist attack is, what the next war is, what the next plague is, we must wonder.

Annabelle Quince: Edwin Black, author of IBM and the Holocaust. My other guests were: Margo Anderson, a professor of history at the University of Wisconsin; Nigel Waters, privacy consultant; and Marc Rotenberg from the Electronic Privacy Information Center.

The sound engineer is Phil McKellar. I'm Annabelle Quince and this is Rear Vision on RN.