Let's talk about performance

It’s no secret, performance on our servers hasn’t been the greatest recently. If you play on or host a popular server, you know we’re not alone in this. Performance across the board has been slipping for months now, and it’s reached critical mass.

What took so long to say something?

We constantly monitor the performance of our servers and are in close communication with the Facepunch developers when we experience issues. Although I’m ever present to this, I don’t make it a habit of harping on it. It is not my intention to bust Facepunch’s balls - I greatly appreciate what they do and fucking love the game they have created. They are always very responsive when we bring things up, and I give them the benefit of the doubt that they will resolve issues in a reasonable amount of time.

I don’t like to leave the community in the dark as to what is going on though. Therefore, I’ve decided to fully flesh out where things are at, what we’re doing about it, what you can do about it, and where we go from here. This focuses on server performance, but we all know client performance could also use some work.

What’s causing this?

A server will run decently for a period of time, then at some point, performance will take a dip, and gameplay becomes either sluggish or completely unplayable. Very rarely does a server recover fully without a restart - and even then, restarts just kick the can down the proverbial road.

There doesn’t appear to be one single cause of these slowdowns. Rather, it’s likely a culmination of various changes and additions to the game over time that has gotten us to this point. Historically, AI has been a main culprit. Between the cargo ship, bandit town, compound, junkpile scientists, animals, and everything else, there is more AI in the game than ever before. The presence of the cargo ship in particular appears to be a catalyst in these dips. It’s not clear if this event is the main cause of the server slowing or simply the straw that's breaking the camel's back.

We also have electricity as a new part of the equation. Although Helk specifically throttled the amount of electrical requests which can be processed by the server at a specific time, there is definitely some overhead involved with this system. All this doesn’t even touch on the shortcomings of the underlying engine of the game, Unity.

In all fairness, Rust is really pushing the boundaries farther than pretty much any other game out there. Each server has potentially hundreds of people at a time gathering, hunting, building, raiding, and the like. It’s really incredible when you think about it, and the sheer massiveness of the scope is one of the things that makes Rust so special.

By the same token, when every action is delayed by seconds, or you end up dying and losing hours of work due to lag, the magic of the game is quickly replaced by anger and frustration. The very things which make the game amazing actually exacerbate the need for optimal performance.

What are we doing to combat this?

As a server owner, this is disheartening to say the least. When complaints of lag flood our Discord, I feel it in the pit of my gut; I cringe when I read posts referencing the poor performance on our servers.

Our Systems Team and I have been doing everything in our power to remedy the situation. We just purchased two new servers with Coffee Lake processors in our US data center. We also rented new hardware in the AU last month so each server in that region now has its own dedicated box. These are not the first times we’ve thrown more hardware at the situation and surely won’t be the last - we are always on the lookout for better hardware to be available from our providers.

As it stands, we run 23 dedicated boxes to host our 31 Rust servers. All of our larger servers have their own dedicated box to run the single instance of Rust. If you have any experience with hosting servers for other games, you know this is fucking insane.

In addition to this, we are running server restarts much more frequently and have set up special alerts to inform our Systems Team when a server’s FPS drops to single digits consistently. We also, as always, are working with the developers at Facepunch to provide them with any information we can to help identify the biggest culprits of low performance.

What should you do?

If you’re playing in our servers and performance drops consistently for 5 minutes or more, let us know in Discord and our Systems Team will look into it ASAP. Please keep in mind that a server restart isn’t a silver bullet and may not always be our immediate course of action.

If you’re looking for a Rust experience that is as smooth as possible, I recommend finding a very low population community server (of which there are numerous great ones). If you only like playing on high pop servers, you may want to consider taking a break from Rust all together till the performance pendulum swings in the other direction.

If you’re a server owner, I suggestion playing with disabling or limiting the amount of AI present on your server as this tends to be one of the largest performance hogs. Unfortunately, this is not an option for Rustafied servers as it fails to qualify as an official vanilla experience.

Where do we go from here?

The ball is in the developer’s court at this point. Thankfully, both Helk and Alistair have told me they are looking into it this month. Will we see some relief in the February update? That remains to be seen, but I sure as fuck hope so.

Thank you for your patience in this matter. I am a strong believer that the best days of Rust are still ahead of us and I am excited to be here, reporting on development and providing the best server experience we possibly can.

Much love,

Bugs