Summary

Between December 2018 and January 2019, the Roblox platform experienced multiple issues with data consistency:

On December 19th and January 8th, many games unexpectedly became private, had their playable device settings modified, or had seemingly random places added to their game without developer action

Between January 7th and January 15th, some games experienced data loss due to Data Stores incorrectly returning nil values for existing entries

Both issues were due to an update to an internal caching library that returned incorrect results from the cache under heavy load. The issue has since been patched.

Resolution

For the January 8th incident, games that were unintentionally made private were restored to their correct state.

In response to Data Stores data loss incident, we have provided backups from before any possible data loss occurred.

Private Games Incident Timeline

December 19th – Several top games became private suddenly and without developer action. We immediately rolled back the change to prevent further damage. We identified and patched a suspected root cause but were not able to confirm whether the issue was resolved because it was not reproducible.

January 8th – Many games unexpectedly became private or have their playable devices changed. At this point we suspected that our earlier patch had not addressed the underlying issue and resumed our investigation.

January 14th – Games that were incorrectly made private on January 8th were restored to their correct state.

January 15th – The root cause is diagnosed and a patch is released. We begin investigating our options for restoring lost data.

Data Store Incident Timeline

January 7th – We started investigating after a developer reported suspected data loss. It was unclear whether this was a game-specific issue.

January 10th – We received several more reports of data loss, confirming that this is a Roblox-wide issue.

January 14th – Developers are informed of the data loss issue and what Roblox is doing to address it.

January 15th – The root cause is diagnosed and a patch is released. We begin investigating our options for restoring lost data.

January 24th – We publicize our plan to provide backups prior to any data loss to developers so they can restore lost data.

February 12th – Data Store backups become available to all developers.

Key Takeaways

We will provide more timely updates about ongoing issues. There were multiple times in the investigation where we failed to publicize relevant information until days later. Going forward, we will strive to close this gap in communication.

The process of creating and exposing Data Store backups took significantly more time than we anticipated. We are investigating what we can do to streamline this process should future incidents arise.

Conclusion

We are striving to provide more transparency when it comes to disruptive incidents so developers have more insight into the issues and what we are doing to address them. Thank you for your patience as we deal with issues like this.