16th Jul 2019

Hi Everyone!

Apologies for the recent server issues/downtime, this should hopefully address it also sorry for the long ass post .

The most recent (yesterday) issue was caused by me testing the waters on one of our masters (Nekomata) in order to make sure the upgrade goes smoothly.

While the upgrade went fine, the server came back up after a reboot, kernel updated, PHP updated to 7.3 etc and was running fine for roughly a week, but it seems a more recent reboot caused Apache to come back up without us noticing.

Previously our servers used to use Apache HTTPd to power s-ul, but very quickly once we grew, we started to see the performance implication and apache not being able to keep up with the requests; as a result this caused us to move to nginx.

However, we never removed the apache2 package from Nekomata; we simply disabled the startup "just in case".

It looks like during the Buster upgrade however it's re-enabled it and as a result, users were presented with the Apache default page for a period.

Second most recent was caused by the PHP7.3 update.

This was similar to a previous outage where the s-ul file cache filled up and basically crumbled.

Why? Because the cronjob pointed to /usr/bin/php7.1 instead of /usr/bin/php

.... and 7.1 had been uninstalled for 7.3, big brain moves right there.

Third most recent was caused by our SSL certificate expiring.

This is your standard "SSL cert expired whoops" kinda outage. Usually I'm on top of this but it slipped by me this month.

Generally LE can auto-renew, but wildcard LE certs require certain DNS updates every time, and as we host our own nameservers and have roughly 7 servers in s-uls infrastructure... it gets a little tricky.

It's something I'm still looking into fixing up automation wise but we're getting there!

The fourth and reoccurring issue is our DNSSEC signatures expire monthly.

This is something I don't really have an excuse for, I keep forgetting to automate this and get sidetracked onto other items.



Another thing I wanted to address is the inactivity/lack of news.

The reason for that is there's not much newsworthy going on behind the scenes, we're all still busy working on other projects mostly!

s-ul is still very much alive though and still my baby, we've currently been thinking of ways we can improve the service and what would make extras/credits more appealing to users as well.

In the coming days I'll be making a news post with a user satisfaction survey (it'll be short, don't worry) and a Discord invite link for support/users to chat.

One thing I didn't address (and should have as it was fixed a month after the problem started), we re-enabled potentially malicious formats.

Now users will be presented with a middleman warning page for certain file formats to allow users to be sure what they're downloading is what they're expecting.

You can see this in action here: https://corin.s-ul.eu/Fw1S02U8

tl;dr stuff went boom because of system upgrades and ssl certs mostly, also I suck at updating you guys.

As always, thanks for sticking around and using s-ul!

- Corin