Get answers from your peers along with millions of IT pros who visit Spiceworks.

Jason Harvey (aka alienth) is the senior sysadmin at a small site you might have heard of called reddit. He's been there for 2.5 years and prior to that he cut his sysadmin teeth at Rackspace, an HR software provider, and an ISP in Alaska. He graciously agreed to be interviewed about what it's like being a sysadmin for such a popular site.

How do you deal with the stress of keeping a site that large up 24/7?

By bottling it all up, ensuring I will explode at some point in the future.

To be honest I don't really take conscious steps to address the stress. While I have been under some extremely stressful circumstances in my career, they don't tend to bother me, and the stress fades quickly with time.

I tend to think that each person will handle stress in their own unique way. You can learn some mitigation methods, but in the end your brain is going to do what it wants.

How do you handle disaster recovery - do you have multiple Amazon Web Services (AWS) sites?

The first step in addressing this is splitting the site across multiple AWS availability zones. That project is still ongoing. Unfortunately there are many pieces of the app that make such a split very difficult (mostly centered around the need for global locking and some layer of a low-latency globally persistent state). Some of those pieces need to be re-factored or completely re-implemented. We're still working towards it, but we have a ways to go.

In the event of a large disaster, such as the AWS US-East region completely going away, the site data can be restored from the regionally redundant S3 backups. This of course incurs some considerable time in which the people will be deprived of a crucial cat-picture outlet.

In the event of a planetary cataclysm, we'll just need to wait a bit for a Poincaré recurrence to occur. I don't expect many complaints will occur in the ensuing downtime.

Who is your role model?

Anyone that I can learn from. I don't really see myself as having one specific role model in mind. Like my hobby fixation, I tend to focus on specific individuals whom I respect and learn as much about them as I can. Throughout the years this has included Linus Torvalds (even though he can be an asshole), Larry Wall, Mike Krahulik and Jerry Holkins (of Penny Arcade fame), and John Carmack, to name a notable few.

Where do you go for ongoing career development? Favorite conferences?

I try to keep tinkering on a personal project or two in order to further my learning. I love finding something technical I know very little about and completely tearing it down till I understand how every bell rings and whistle blows.

I'm not much of a tech-conference goer. The one that sticks out in my mind as the most enjoyable is PgCon.

Biggest disaster? Biggest triumph?

I have yet to have caused any major disasters in my career (knock on wood), although I know that day will inevitably dawn. There was an incident where I was adjusting the mount flags on a slew of systems and mistakenly mounted filesystems that were in an active MySQL RHCS cluster. This resulted in some MySQL data becoming irrecoverably corrupted. While I wouldn't characterize it as a disaster, it did sting considerably.

Biggest triumph has probably been getting reddit to a more stable state. When I joined, the infrastructure was in considerable trouble. The state of things was of no fault of the existing team; they simply lacked the resources and time to address technical debt. There was a long period of time where we had 8+ hours of downtime *a week*.

With the help of the dev team (which at the time consisted of one person) and a lot of lost sleep, I managed to get things to a relatively stable state during my first year here. We still have a long way to go on stability, but we've made considerable progress since the outage-filled months of 2011.

What is the most satisfying thing about being a reddit sysadmin?

There are a few aspects of the position which are very satisfying. First being that I work with people who are extremely talented, respectful of each other, and dedicated to helping one another. If there is a site issue, we don't waste time assigning blame (except maybe in jest), we simply work as a team to get it fixed. Places that I've worked at in the past have often had an environment where the development and sysadmin team were each in their own respective silos. After experiencing how things work at reddit I can't imagine going back to such an environment.

Another amazing thing about reddit is that we are in direct control of the future of the site. As a team, we make the calls on how the site should be run from a technical, community, and business perspective. There are obviously scenarios where an individual will need to take the responsibility of making a call, but even then those calls are discussed with the entire team first. Of course, being in control of our destiny means that we are also directly and wholly responsible for our failures.

To top it off, the freedom which we are given to work on whatever we can help with is extremely refreshing. My main responsibility is focused on systems administration, but in the 2.5 years here I've worked on addressing community issues, helping develop the site privacy policy, arranging community charity marathons, and writing official blog posts which analyzed legislative text. I've worked on these things not because I was forced to, but because I saw areas where I could be helpful and I was given the freedom to work on them. That same freedom is given to any member of the team.

What is the worst thing about being a reddit sysadmin?

The most frustrating things that I work with aren't really on the sysadmin side, but more on general site management. Dealing with problematic, dangerous assholes is probably the worst thing. There is a category of users who will spend as much time as possible trying to cause as much damage as possible. Dealing with them is a constant arms race that takes up a lot of time, and ignoring them is not an option due to the damage they could potentially cause.

How did you personally feel about the Boston bombing situation and how the reddit community responded?

Different portions of the community responded very differently. I feel a lot of the activity on the site was positive. The site served as a place for people to gather and discuss the events. In the confusion a lot of people desperately wanted to understand the events, and learn if there was any way they could be of help.

Of course, as most people are aware there were some very negative results as well. My hope is that in the future the community as a whole will be more cautious about handling those type of situations. If anything the negative consequences should serve as a lesson to both us and the community.

Current favorite subreddit?

I've been enjoying http:/ / reddit.com/ r/ redditdayof a lot lately. Each day the subreddit focuses on a new topic, and people link to information about that topic.

What other sites or communities do you hang out on besides reddit?

Being a fan of StarCraft I spend a lot of time lurking the TeamLiquid boards. I also bounce around in Hacker News from time to time. Most of my non-reddit time on the internet circles around looking through my RSS feeds, and chilling out in various IRC channels.

What are your non-IT hobbies?

I spend a good chunk of time reading and gaming. I tend to heavily fixate on one specific non-IT thing for a month or so, and then move on to something else. Lately my fixation has been on sci-fi. I just finished reading Hyperion, which I found very enjoyable.

Favorite video game of all time - StarCraft or something else?

It's hard for me to name one specific game, since I feel like so many of them are associated with different experiences. StarCraft is probably my favorite competitive game, even though I'm fairly terrible at it. My recent favorite game is Bioshock Infinite.

Peanut butter: smooth or crunchy?

When I was a child my feet were firmly planted in Camp Crunchy. Since then, I have shirked my childish ways and now prefer smooth.

--

Thank you Jason! If anyone has other questions for Jason feel free to post in the comments or consult some past AMAs he and his fellow sysadmin Ricky Ramirez (aka rram) have done on reddit.

Have a notable IT pro you'd like to see an interview with? Shoot me a PM and I'll see if I can interest them in an interview. In the meanwhile, check out our past interviews with other notable IT pros.