Privacy matters, because someday our personal data might be making choices for us. This was the argument of Kate Crawford, a principal at Microsoft Research speaking at MIT’s EmTech conference Wednesday. Currently, there is scarcely any regulation of data scraped from our social media presences and online activity, and companies’ use of that information could result in discrimination. Craig Mundie, a senior adviser to the CEO at Microsoft, said while speaking at the same conference that this misuse of personal data is so bad it ought to be classified as a felony.

Websites are snatching and grabbing data any way they can nowadays, both to improve their own services and also to anonymize and sell. The trouble is that anonymization rarely works; while big data is often touted as a way to identify larger trends or arcs, it’s almost trivially easy to re-individualize and use on one person.

As a result, companies are increasingly using personal identifying information collected online to shape the experience of their customers. But it’s not just for harmless personalization: big data is “being used for more and more precise forms of discrimination—a form of data redlining,” said Crawford.

Redlining is an old practice that involves denial of service, or the offering of different services, at institutions like banks or healthcare companies based on a profile the customer might fit, such as if they are black or a woman or live in a low-income neighborhood. While discriminatory big data is being used for the same ends as redlining, it doesn’t fit the technical definition. Therefore, its use creates a loophole.

Per an article Crawford cited from Scientific American, redlining specifically prohibits companies from differing their offers to potential customers based on what they know about them. For instance, they can’t offer a loan at six percent interest to someone who lives in Tonyville, CA and then turn around and offer the same loan at 12 percent interest to someone who lives in Boondoggle, TX.

In the old days, if both of those customers saw the six percent offer on a billboard and walked in the bank to apply, the bank couldn’t legally offer the customer from the bad neighborhood the higher interest rate based on where they live. But with online targeting, the bank can make sure the bad-neighborhood customer never sees the offer, period, avoiding the perceived “risk” altogether.

“You cannot know the assessments that are being made about you,” said Crawford. Mundie echoed her concerns Thursday, saying, “Increasingly, the data you should be worried about, you don’t even know about.”

Crawford cited some examples that fall to onto the barely-regulated end of the spectrum: a WebMD search about breast cancer plus a book you buy from Amazon about cancer survival, mixed with a few tablespoons of anonymization and a sprinkling of cross-referenced geographical/demographic data to negate that anonymization, yields you unable to get health insurance or approved for a loan. Why? Data indicates you might possibly be dead soon. It doesn’t matter so much that the breast cancer search and book were out of curiosity and generosity, respectively, for your dying great aunt who has never touched the Internet.

That is a severe example, and it represents a set of decisions that likely wouldn’t be made without more information. But as Crawford pointed out, personal data collected and anonymized and resold is not subject to HIPAA, or very much privacy protection at all, beyond the anonymization most companies offer.

Crawford doesn’t suggest that we do away with anonymization. Rather, she argues that the tactic is being overworked—it’s not providing the protection needed. In a related paper, Crawford describes the need for a “procedural data due process.” Namely, this would involve requiring that companies disclose the information being used for or against someone and the reasoning for it. Disclosure and information processing would also have to be regulated by an unbiased party.

Crawford cited the infamous example of a woman whom Target determined was pregnant before her family did, saying it “will simply look quaint compared to what’s coming down the pipeline.” Mundie suggests that consumers need more control over not only how their data is being used, but how companies will decide to use it in a future. He believes companies that violate the stated terms of use need to face harsher punishments. "Make it a felony,” Mundie said. “Otherwise, the penalty is too low to deter that behavior.”