Facebook has mined a lot of data about its users over the years—relationships, political leanings, and even phone call logs. And now it appears Facebook may have inadvertently extracted another bit of critical information: users' login credentials, stored unencrypted on Facebook's servers and accessible to Facebook employees.

Brian Krebs reports that hundreds of millions of Facebook users had their credentials logged in plain text by various applications written by Facebook employees. Those credentials were searched by about 2,000 Facebook engineers and developers more than 9 million times, according to a senior Facebook employee who spoke to Krebs; the employee asked to remain anonymous because they did not have permission to speak to the press on the matter.

In a blog post today, Facebook Vice President of Engineering, Security, and Privacy Pedro Canahuati wrote that the unencrypted passwords were found during "a routine security review in January" on Facebook's internal network data storage. "This caught our attention because our login systems are designed to mask passwords using techniques that make them unreadable. We have fixed these issues and, as a precaution, we will be notifying everyone whose passwords we have found were stored in this way."

Canahuati noted that the passwords were never visible to anyone outside Facebook and that there was "no evidence to date that anyone internally abused or improperly accessed them... We estimate that we will notify hundreds of millions of Facebook Lite users, tens of millions of other Facebook users, and tens of thousands of Instagram users."

Facebook Lite is a version of the mobile Facebook application "predominantly used by people in regions with lower connectivity," as Canahuati put it. The Android app is most popular in Brazil, Mexico, India, Indonesia, and the Philippines, as well as other countries in South Asia with older 2G and 3G GSM networks—markets where Facebook has experienced much of its recent growth. Lite uses a proxy architecture, with an application server running most of the application code and minimizing the amount of data that needs to be sent to the user's phone. And apparently because it was acting as a proxy, the server was acting on behalf of users and logging their credentials for use in connecting to other Facebook services.

While Facebook Lite users make up the vast majority of those affected, other applications were clearly also involved—as Instagram and non-Lite Facebook accounts were also logged. Canahuati said that Facebook's server-side applications are only supposed to store a "hashed" mathematical representation of users' passwords and not the passwords themselves. But some applications within the Facebook and Instagram architecture clearly didn't do that. According to the Krebs report, the unprotected passwords were stored at least since 2012 until January of this year, when the issue was "discovered".

According to Krebs' source at Facebook, the company may be artificially reducing the size of the possible exposure of passwords. "The longer we go into this analysis, the more comfortable the legal people are going with the lower bounds [of potentially affected users]," the source said. "Right now, they're working on an effort to reduce that number even more by only counting things we have currently in our data warehouse."

Canahuati offered the usual advice for users concerned about their privacy:

You can change your password in your settings on Facebook and Instagram. Avoid reusing passwords across different services.

Pick strong and complex passwords for all your accounts. Password manager apps can help.

He also mentioned use of other features Facebook offers to prevent someone from using stolen user credentials to log in to its services—including two-factor authentication (2FA) through the mobile application or via text message, or the use of a USB security key. But these authentication methods may not be easily available to or effective for many of those affected by this or other password exposures. Using SMS-based 2FA over 2G networks with weak encryption doesn't seem ideal, and thanks to Facebook's use of phone numbers to find profiles, connecting a phone number with a Facebook username is fairly simple.