Summary Socorro is the crash ingestion pipeline for Mozilla's products like Firefox. When Firefox crashes, the Breakpad crash reporter asks the user if the user would like to send a crash report. If the user answers "yes!", then the Breakpad crash reporter collects data related to the crash, generates a crash report, and submits that crash report as an HTTP POST to Socorro. Socorro saves the crash report, processes it, and provides an interface for aggregating, searching, and looking at crash reports. 2018q3 was a busy quarter. This blog post covers what happened.

Staff changes in 2018q3 At the beginning of 2018q3, there were two engineers and two interns. Ced and Alexis ended their internship in August. Mike got slid over to a tiger team which he'll be on until November. For most of September, it was just me. Here's a handy table showing engineering staff over time: Month Engineering Team July Mike, Alexis, Ced, me August Mike (part time), Alexis (until August 10th), Ced (until August 10th), me September me One person isn't a critical mass for a project, so we had to make changes in development process and priorities. It's been tough. If you want to talk more about that, toss me an email, ping me on IRC, or catch me at the All Hands in December.

Starting a monthly newsletter I've been solo on other projects at Mozilla before. This is the first time I've been the solo engineer with an amorphous mass of stakeholders. When I worked on Input, we had weekly meetings to work out what was going on and for me to get feedback on engineering work and for other people to learn about upcoming changes. With Socorro, it's really different. I don't have a good feel for who the stakeholders are, what they're doing, what their current/future plans are, or whether they find anything I'm doing helpful or not. At first it was just inconvenient because it made it hard to prioritize things. Now I find it somewhat demoralizing. I feel like it's just me and Wilson on an island somewhere. Other projects are doing monthly blog post updates. Maybe if I do that, it'll help. This is a 2018q3 rollup. After this, I'll start doing monthly updates.

Highlights in 2018q3 We accomplished a lot in 2018 quarter 3. Lots of improvements to Socorro-style signature generation: Extracted Socorro signature generation into a standalone library. See my blog post on socorro-siggen. Fixed issues with C++ prefixes and types. Added handling for C++ cv and ref qualifiers. Built better support for Rust frames. Improved documentation for signature generation.

Reworked Crash Stats to support new products that don't follow the same build methodologies as Firefox. That work allowed us to add GeckoView, Focus and FirefoxReality.

Changed the throttling code to reduce the number of crash reports coming into Socorro. Socorro no longer accepts crash reports for unsupported products. Socorro no longer accepts crash reports that it doesn't plan to process. Previously, Socorro was saving roughly 70% of incoming crashes. Now it's saving roughly 20% and all those crashes are searchable in the Crash Stats interface. Socorro is no longer saving crash data that we don't know anything about.

Redesigned the about:crashes page to fix a bunch of UI/UX issues.

Removed ADI and all ADI-requiring things.

Improvements to make Crash Stats more service-y: Reworked Crash Stats user documentation. Added documentation for Memory Dump and PII access. Added documentation for adding new products.

Added a bunch of fields to supersearch.

Removed PostgreSQL from the processor. This is a big deal because it was one of the things blocking a processor rewrite.

Audited who had access and removed access from a bunch of people. Auditing now happens monthly.

Made a lot of progress with transitioning to Python 3. Linting is fine. Fixing code is ongoing. Dropped some libraries that don't support Python 3 or were unmaintained. Vendored Crontabber.

Redid Docker infrastructure so we have a single Docker image that all the services use. This simplified a lot of things and makes Socorro development easier.

Thank you! Thank you to Alexis and Ced! I hope you got as much out of your internship experiences as you could have. Thank you for being part of the team and the work you did! Thank you to Ted, Kartikaya, Andrew, Jeff, Nicholas, Nathan, and others for helping to guide fixing signature generation to work better with Rust! Thank you to Chris, Snorp, and others for helping to guide new product support on Socorro! Thank you to Marcia, Ted, Liz, Philipp, dbaron, Gabriele, and others for helping to guide the reduction in crash report accepting work!