Microsoft has pulled its latest Windows 10 update offline after some users complained of missing files. It’s the latest in a string of incidents with regular patches and Microsoft’s larger Windows 10 updates that have been causing issues for some PC users this year. While Microsoft tests Windows 10 with millions of beta testers, there are signs that this public feedback loop isn’t always working. Earlier this year Microsoft delayed its April 2018 Windows 10 update due to last minute Blue Screen of Death issues, and then had to fix desktop and Chrome freezing issues after it was shipped to more than 600 million machines.

Microsoft now faces questions over how these updates have caused big issues, and why the company didn’t pick them up in testing. These questions are especially relevant as it appears Microsoft was warned about both of these major bugs before the company shipped the April and October 2018 updates. Reports of the desktop freezing bug were submitted multiple times by testers earlier this year, but don’t appear to have been flagged as a bigger problem because they weren’t up voted.

Microsoft was warned over these issues

Likewise, the recent data deletion issue was flagged in feedback reports from months before Microsoft released the October 2018 Update last week. It’s still not clear how many are affected by this current issue, but it’s enough to have forced Microsoft to pull the update entirely — an unusual step for the company.

Microsoft’s big change for Windows 10 was listening to its customers after the Windows 8 disaster. Instead of developing the operating system behind closed doors, the company opened it up for everyone to test early access to builds and help report issues. It was a daunting prospect for then-Windows chief Terry Myerson, who admitted in a 2015 interview with The Verge that “you’re putting it out there when it’s not done, then you’re getting all kinds of feedback and stuff that you know is broken.”

Microsoft may have been relying on its Windows Insider program too much for Windows 10, though. Microsoft largely phased out its dedicated Software Test Engineer (STE) roles for Windows during a huge round of layoffs a year ahead of the Windows 10 release. Instead, it has favored developers testing their own work, or reports from the Windows Insider feedback program.

Hal Berenson, who spent years working at Microsoft as a distinguished engineer, believes there are three possibilities for this latest data deletion bug shipping to the public. “(1) They couldn’t isolate the problem and decided it was rare enough to ship anyway. (2) Automated filtering tools failed to catch that this was a serious issue despite rarity. (3) They put in a fix, but it didn’t fix all cases,” says Berenson in tweet. “My actual vote is on #3, because I’ve seen that happen many times in my career.”

Microsoft has not yet revealed exactly why this deletion bug made it into the Windows 10 October 2018 Update, but it’s unlikely the company ever will. A support article reveals Microsoft is investigating “isolated reports” of documents going missing after the Windows 10 October 2018 Update is installed. Microsoft’s Windows Insider chief, Dona Sarkar, says affected users should call Microsoft’s support line as the company has “the tools to get you back to a good state.”

Either way, it’s not a good look for Microsoft’s Windows 10 feedback program. Microsoft was bold in its move to allow anyone to test Windows 10, but it now needs to recognize some of these issues with Windows software quality. Windows 10 is also facing a number of issues related to regular monthly security update patches, and those even forced enterprise patching veteran Susan Bradley to write an open letter to Microsoft earlier this year. The company’s response to these issues is now a big test for Windows 10, which has been generally well received. If Microsoft is truly listening to customers then now is the time to prove it.