A few months ago, there was the post on Reddit of an entry level developer who deleted the production database on his first day. We all cringe reading stories like this of those who make those big, unforgettable mistakes. We realize it wouldn’t take much for that to be us—most have had close calls.

In my first job, a senior database administrator dropped the production database on his first day. These stories are everywhere. The team restored his mistake from a week old backup and kept him around. Ten years later they still poked fun at him for it.

One morning earlier this year I was called on to look into a production problem for a client. They were beginning to beta test their site with a small audience when overnight their site’s homepage didn’t have anything on it. I wondered if there was a bug or vulnerability that led to this.

I signed in to the production machine and pulled up the database. The articles table was empty. OK, that confirmed what we were seeing on the website.

The users table still had users. Weird. So we lost all our articles but at least their beta users still had their accounts. We could explain that it’s a beta and these things happen.

The next few moments were a blur. I don’t recall exactly what I did. I don’t think I was dumb enough to type drop table users in the console. But there I was, now with no articles and no users table. I sat there in shock for a bit.

Then my mind raced on how to fix this. Did I really drop the users table? Yes. Did we run backups? No. How do we tell the client this? I don’t know.

I remember walking over to the project manager, sitting down next to her, and explaining what had happened. We didn’t have data in our articles table, so that’s why the site looked empty. And oh yeah, I also dropped the users table. They were now going to need to re-invite all those users—if they could figure out who they all were. Yikes.

I went back to my desk feeling defeated.

Something didn’t sit right with me, though. How did we lose all those articles in the first place?

I kept digging. Part denial, part wanting to save face. Shortly afterwards, I noticed something important.

There were five other databases on the server. One of them had a name similar to the database I had just been looking at.

When I checked it out, all the articles were there. The users table was fine. It turns out a configuration change had inadvertently made it to production, causing the site to point to a brand new database. Those users I saw? Seed data.

What a relief! A morning of nerves and stomach acid making me feel sick, but we were able to “recover” all the data and I had found the real issue before we were to communicate the bad news.

Lots of lessons learned from the episode. One of the simplest: now we always do backups… perhaps a developer’s most effective antacid.

Rushing And Never Getting Ahead