For the last month I've been running a middle relay (no guard flag yet) on a 512 MB VPS provided by Edis.at in Switzerland (4.99 euro per month). The software is Tor 0.2.5.8-rc with obfsproxy 2 and 3 running on up-to-date Debian Wheezy 7.6. I left the iptables at their defaults.
Last night Edis suspended my VPS because of a suspected outgoing DDoS attack, and emailed me. I rang their (very helpful) phone support in London and they de-suspended the VPS so I could ssh into it and investigate. ( I told them it was running a tor non-exit relay - that wasn't a problem.)
Sure enough the Solus control panel traffic graphs show the tor relay traffic stopping abruptly last night, followed about 40 minutes later by an exponentially increasing spike of outgoing traffic, soon cut off when the VPS was suspended.
I ssh'd into the VPS ("last login" showed my own last login, so no problem there) and looked at logs and top. notices.log simply cut off when the tor traffic stopped, but showed nothing unusual before that.
The Solus control panel traffic graph started showing (a very small amount of) outgoing traffic as soon as the VPS was de-suspended, so I assumed the malware was still active, and used top.
This is typical of what I found.
1 root 20 0 10604 832 700 S ... 0:00.10 init 2 root 20 0 0 0 0 S ... 0:00.00 kthreadd/3277 3 root 20 0 0 0 0 S ... 0:00.00 khelper/3277 1370 root 20 0 36976 660 492 S ... 0:00.38 hxyqbutesc 1400 root 20 0 109m 1616 1188 S ... 0:00.00 rsyslogd 1446 root 20 0 18836 884 680 S ... 0:00.00 cron 1473 root 20 0 49888 1212 608 S ... 0:00.00 sshd 3187 root 20 0 27556 744 504 S ... 0:00.00 vzctl 3188 root 20 0 17808 1968 1500 S ... 0:00.00 bash 5374 root 20 0 21584 1416 1072 R ... 0:00.00 top 5440 root 20 0 6244 536 296 S ... 0:00.00 akcviaxtbl 5443 root 20 0 6244 532 296 S ... 0:00.00 akcviaxtbl 5446 root 20 0 6244 532 296 S ... 0:00.00 akcviaxtbl 5448 root 20 0 6244 536 300 S ... 0:00.00 akcviaxtbl 5449 root 20 0 6244 532 296 S ... 0:00.00 akcviaxtbl
There is a strange line near the top for process "hxyqbutesc", which didn't change; and a strange block of lines at the bottom, which changed every second or two. Sometimes it was five similar lines, as above; sometimes it was several block of lines, eg this:
5276 root 20 0 6244 528 292 S 0.0 0.1 0:00.00 qscntoweqb 5277 root 20 0 6244 536 300 S 0.0 0.1 0:00.00 qscntoweqb 5280 root 20 0 6244 532 296 S 0.0 0.1 0:00.00 qscntoweqb 5282 root 20 0 6244 536 300 S 0.0 0.1 0:00.00 qscntoweqb 5283 root 20 0 6244 528 292 S 0.0 0.1 0:00.00 qscntoweqb 5290 root 20 0 6244 532 296 S 0.0 0.1 0:00.00 saqizxaihz 5294 root 20 0 6244 528 300 S 0.0 0.1 0:00.00 saqizxaihz 5295 root 20 0 6244 532 296 S 0.0 0.1 0:00.00 saqizxaihz 5296 root 20 0 6244 524 296 S 0.0 0.1 0:00.00 saqizxaihz 5298 root 20 0 6244 528 296 S 0.0 0.1 0:00.00 saqizxaihz
Each block is always 5 lines, and the names (always 10 lower-case letters) seem to be different every time. The blocks change fairly regularly every second or two.
I shut the VPS down to stop it doing any more harm, but I didn't delete anything; I can restart it and ssh in again for further investigation if necessary.
Eventually I'll have to reinstall everything from scratch, straightforward enough, but what can I do to make sure it doesn't happen again? Would hardening my iptables work? Has anyone else seen this?
Nick Sheppard
On 2014-10-25 16:36, Nick Sheppard wrote:
For the last month I've been running a middle relay (no guard flag yet) on a 512 MB VPS provided by Edis.at in Switzerland (4.99 euro per
You are aware that Edis is considered a bit shady I hope; though this is likely because they attract a lot of cheap customers and thus get a lot of abuse out of their network. The question becomes at one point if it is a separate customer or themselves though.
Also note that they are just reselling other peoples services, hence why they are cheap as they oversubscribe a lot.
The Solus control panel traffic graph started showing (a very small
That Solus control panel could have been a way in to your system.
What kind of virtualization is used?
[..]
Each block is always 5 lines, and the names (always 10 lower-case letters) seem to be different every time. The blocks change fairly regularly every second or two.
The virus/bot/etc is respawning processes so that nobody can easily kill them.
pstree will show you where the process comes from originally.
The random name makes classification easier.
[..]
Eventually I'll have to reinstall everything from scratch, straightforward enough, but what can I do to make sure it doesn't happen again? Would hardening my iptables work? Has anyone else seen this?
Actually secure a machine.
Most likely they just guessed your password by an automated SSH login botnet.
Using SSH keys, firewalling SSH off except for trusted hosts and not having any services listening that you do not want are key to a properly secured system. There are lots of articles on the interwebz about it.
Greets, Jeroen
Hey Nick,
Some general advice for securing a VPS;
- disable root ssh login - disable ssh password login and only allow key based logins (https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2) - install fail2ban - disable / remove any services you aren't using - use full disk encryption - use a decent iptables config (https://www.linode.com/docs/security/securing-your-server#creating-a-firewal...) - keep the box up to date using unattended upgrades (https://wiki.debian.org/UnattendedUpgrades)
Generally fail2ban, key-only login, iptables, and software updates should be enough to keep the box secure. You probably got hacked either through pwd bruteforcing (which would be prevented by fail2ban or disabling pwd login) or by a vulnerability (which would be prevented by software upgrades).
I've generally found Edis to be very good (e.g. one of my VPS has an uptime > 1 year).
Cheers,
Dan
On 25/10/14 16:36, Nick Sheppard wrote:
For the last month I've been running a middle relay (no guard flag yet) on a 512 MB VPS provided by Edis.at in Switzerland (4.99 euro per month). The software is Tor 0.2.5.8-rc with obfsproxy 2 and 3 running on up-to-date Debian Wheezy 7.6. I left the iptables at their defaults.
Last night Edis suspended my VPS because of a suspected outgoing DDoS attack, and emailed me. I rang their (very helpful) phone support in London and they de-suspended the VPS so I could ssh into it and investigate. ( I told them it was running a tor non-exit relay - that wasn't a problem.)
Sure enough the Solus control panel traffic graphs show the tor relay traffic stopping abruptly last night, followed about 40 minutes later by an exponentially increasing spike of outgoing traffic, soon cut off when the VPS was suspended.
I ssh'd into the VPS ("last login" showed my own last login, so no problem there) and looked at logs and top. notices.log simply cut off when the tor traffic stopped, but showed nothing unusual before that.
The Solus control panel traffic graph started showing (a very small amount of) outgoing traffic as soon as the VPS was de-suspended, so I assumed the malware was still active, and used top.
This is typical of what I found.
1 root 20 0 10604 832 700 S ... 0:00.10 init 2 root 20 0 0 0 0 S ... 0:00.00 kthreadd/3277 3 root 20 0 0 0 0 S ... 0:00.00 khelper/3277 1370 root 20 0 36976 660 492 S ... 0:00.38 hxyqbutesc 1400 root 20 0 109m 1616 1188 S ... 0:00.00 rsyslogd 1446 root 20 0 18836 884 680 S ... 0:00.00 cron 1473 root 20 0 49888 1212 608 S ... 0:00.00 sshd 3187 root 20 0 27556 744 504 S ... 0:00.00 vzctl 3188 root 20 0 17808 1968 1500 S ... 0:00.00 bash 5374 root 20 0 21584 1416 1072 R ... 0:00.00 top 5440 root 20 0 6244 536 296 S ... 0:00.00 akcviaxtbl 5443 root 20 0 6244 532 296 S ... 0:00.00 akcviaxtbl 5446 root 20 0 6244 532 296 S ... 0:00.00 akcviaxtbl 5448 root 20 0 6244 536 300 S ... 0:00.00 akcviaxtbl 5449 root 20 0 6244 532 296 S ... 0:00.00 akcviaxtbl
There is a strange line near the top for process "hxyqbutesc", which didn't change; and a strange block of lines at the bottom, which changed every second or two. Sometimes it was five similar lines, as above; sometimes it was several block of lines, eg this:
5276 root 20 0 6244 528 292 S 0.0 0.1 0:00.00 qscntoweqb 5277 root 20 0 6244 536 300 S 0.0 0.1 0:00.00 qscntoweqb 5280 root 20 0 6244 532 296 S 0.0 0.1 0:00.00 qscntoweqb 5282 root 20 0 6244 536 300 S 0.0 0.1 0:00.00 qscntoweqb 5283 root 20 0 6244 528 292 S 0.0 0.1 0:00.00 qscntoweqb 5290 root 20 0 6244 532 296 S 0.0 0.1 0:00.00 saqizxaihz 5294 root 20 0 6244 528 300 S 0.0 0.1 0:00.00 saqizxaihz 5295 root 20 0 6244 532 296 S 0.0 0.1 0:00.00 saqizxaihz 5296 root 20 0 6244 524 296 S 0.0 0.1 0:00.00 saqizxaihz 5298 root 20 0 6244 528 296 S 0.0 0.1 0:00.00 saqizxaihz
Each block is always 5 lines, and the names (always 10 lower-case letters) seem to be different every time. The blocks change fairly regularly every second or two.
I shut the VPS down to stop it doing any more harm, but I didn't delete anything; I can restart it and ssh in again for further investigation if necessary.
Eventually I'll have to reinstall everything from scratch, straightforward enough, but what can I do to make sure it doesn't happen again? Would hardening my iptables work? Has anyone else seen this?
Nick Sheppard
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Sat, Oct 25, 2014 at 03:36:05PM +0100, Nick Sheppard wrote:
This is typical of what I found.
1 root 20 0 10604 832 700 S ... 0:00.10 init 2 root 20 0 0 0 0 S ... 0:00.00 kthreadd/3277 3 root 20 0 0 0 0 S ... 0:00.00 khelper/3277 1370 root 20 0 36976 660 492 S ... 0:00.38 hxyqbutesc
I should note here that yes indeed, you do appear to have been compromised.
We get some relay operators here who misinterpret an email from their ISP, which tells them they've been compromised but really the only evidence is that they sent out some traffic that the other side thought could only have been sent if they're compromised. E.g., https://lists.torproject.org/pipermail/tor-relays/2014-October/005551.html
But this one does not look good. I sense a reinstall in your future. :)
Eventually I'll have to reinstall everything from scratch, straightforward enough, but what can I do to make sure it doesn't happen again? Would hardening my iptables work? Has anyone else seen this?
The other advice I heard here was very good too -- mainly "be sure to do all your updates", "don't allow ssh login by password", and "wonder if perhaps your hosting provider has a problem that makes it impossible for you to keep your host safe".
--Roger
First, many thanks for all the advice. I will do some serious homework.
To the question What kind of virtualisation? It's OpenVZ. In fact it's the OVZ Starter option from here:
http://www.edis.at/en/server/linux-vps-solusvm-openvz/zurich
and the OS option I chose is what Edis describe as "Debian 7 amd64 minimal (64bit)"
I've restarted the VPS for 5-10 minutes at a time, long enough to watch the malware at work but not long enough for it to ramp up its flood of output. It seems to live here, in /boot:
root@xxxxx:/boot# ls -al total 128 drwxr-xr-x 2 root root 4096 Oct 26 14:19 . drwxr-xr-x 23 root root 4096 Oct 26 13:58 .. -rwxr-xr-x 1 root root 122698 Oct 24 17:19 hxyqbutesc
and it's started in a straightforward way from /etc and /etc/init.d ; rc1.d, rc2.d and so on have links to /etc/init.d/hxyqbutesc :
root@xxxxx:/etc# ls -l rc1.d total 4 1 root root 18 Sep 13 20:21 K01quotarpc -> ../init.d/quotarpc 1 root root 13 Sep 14 22:41 K01tor -> ../init.d/tor 1 root root 17 Sep 13 20:21 K04rsyslog -> ../init.d/rsyslog 1 root root 369 Aug 5 07:47 README 1 root root 18 Sep 13 20:21 S01bootlogs -> ../init.d/bootlogs 1 root root 20 Oct 24 17:19 S01hxyqbutesc -> ../init.d/hxyqbutesc ...
root@xxxxx:/etc# ls -l rc2.d total 4 1 root root 677 Aug 5 07:47 README 1 root root 18 Sep 13 20:21 S01bootlogs -> ../init.d/bootlogs 1 root root 20 Oct 24 17:19 S01hxyqbutesc -> ../init.d/hxyqbutesc ...
and in /etc/init.d is a short script:
root@xxxxx:/etc/init.d# ls -l total 196 -rw-r--r-- 1 root root 2427 Aug 5 07:47 README -rwxr-xr-x 1 root root 1276 Aug 5 07:47 bootlogs -rwxr-xr-x 1 root root 1281 Aug 5 07:47 bootmisc.sh -rwxr-xr-x 1 root root 3816 Aug 5 07:47 checkfs.sh -rwxr-xr-x 1 root root 1099 Aug 5 07:47 checkroot-bootclean.sh -rwxr-xr-x 1 root root 9673 Aug 5 07:47 checkroot.sh -rwxr-xr-x 1 root root 3033 Aug 5 07:47 cron -rwxr-xr-x 1 root root 1329 Aug 5 07:47 halt -rwxr-xr-x 1 root root 1423 Aug 5 07:47 hostname.sh -rwxr-xr-x 1 root root 3880 Aug 5 07:47 hwclock.sh -rwxr-xr-x 1 root root 317 Oct 26 13:58 hxyqbutesc ...
which looks like this:
root@xxxxx:/etc/init.d# cat hxyqbutesc #!/bin/sh # chkconfig: 12345 90 90 # description: hxyqbutesc ### BEGIN INIT INFO # Provides: hxyqbutesc # Required-Start: # Required-Stop: # Default-Start: 1 2 3 4 5 # Default-Stop: # Short-Description: hxyqbutesc ### END INIT INFO case $1 in start) /boot/hxyqbutesc ;; stop) ;; *) /boot/hxyqbutesc ;; esac
and runs the main file /boot/hxyqbutesc , which is mainly hex numbers with many embedded filenames and paths. The timestamp Oct 24 17:19 is an hour or two before the VPS was suspended on Friday evening (can't be more precise yet because I'm not sure what timezone the suspension email was sent from).
Thank you to the people who suggested using ps as well as top. ps -x gives eg this when the malware is working:
PID TTY STAT TIME COMMAND 1 ? Ss 0:00 init [2] 2 ? S 0:00 [kthreadd/3277] 3 ? S 0:00 [khelper/3277] 1373 ? Ssl 0:00 who 1406 ? Sl 0:00 /usr/sbin/rsyslogd -c5 1451 ? Ss 0:00 /usr/sbin/cron 1478 ? Ss 0:00 /usr/sbin/sshd 1857 ? Ss 0:00 vzctl: pts/0 1858 pts/0 Ss 0:00 -bash 3474 ? Ss 0:00 ls -la 3477 ? Ss 0:00 cat resolv.conf 3478 ? Ss 0:00 route -n 3479 ? Ss 0:00 ifconfig eth0 3481 ? Ss 0:00 su 3489 ? Ss 0:00 sleep 1 3492 ? Ss 0:00 ifconfig 3493 ? Ss 0:00 sh 3494 ? Ss 0:00 grep "A" 3496 ? Ss 0:00 pwd 3497 pts/0 R+ 0:00 ps -x
For process 1373 top would give "hxyqbutesc" instead of "who" (this line doesn't change), and the ten lines for 3474 - 3496 would be the two changing blocks of ten-letter random names. Assuming the glimpses of command names here can be taken at face value, something very sinister is clearly going on ... other commands that flash into view are "who, whoami, uptime, bash, id, cd /etc, netstat -antop, top, echo "find", gnome-terminal".
As for how it got in, most people seem to suspect an attack through ssh. But when I ssh'd in yesterday to start investigating, I was careful to note the "Last Login" message, and it gave my own genuine last login, three weeks ago from my home IP. Can I take this at face value to mean that ssh has not been used between my own two logins? Can "Last Login" be falsified?
Some suggest an exploit using out-of-date software. Well, I last did an apt-get dist-upgrade 29 days ago, (after the bash fix), and tor and obfsproxy (included in the dist-upgrade) were the only things I installed over the "minimal" Debian 7.6 - so I was reasonably up-to-date.
And some suggested that the malware might have come through the Solus control panel (which of course has root access - I ssh in by requesting a temporary (time-limited) console from Solus, and it gives me a one-time strong password to use for that). Is this plausible? perhaps the next step is to send all this to Edis support and see if they noticed anything unusual around 17:19 last Friday?
Thanks again for everybody's help,
Nick Sheppard
Hello Nick, I hop you don't mind a few pointers on this based on my experience of hacked sites: When listing directories, use 'ls -alct' to show hidden files as well, and the ctime rather than the mtime - mtime is trivial to falsify. When using 'ps', compare the process names with those given by running (as root) 'lsof -p <processnum>' where <processnum> is the number of the suspect process. The entries with 'txt' and 'cwd' in the fourth column will let you see the files connected to the process, which can be useful if a process is spoofing its name or the file that was run was deleted by the process to try to cover its tracks. Entries with 'IPv4' in the fifth column will show any network connections that processes have opened up (visible on their own using 'lsof -i') in case the bot is trying to call home. Regards, Geoff
On 26/10/14 19:46, Geoff Down wrote:
Hello Nick, I hop you don't mind a few pointers on this based on my experience of hacked sites: When listing directories, use 'ls -alct' to show hidden files as well, and the ctime rather than the mtime - mtime is trivial to falsify. When using 'ps', compare the process names with those given by running (as root) 'lsof -p <processnum>' where <processnum> is the number of the suspect process. The entries with 'txt' and 'cwd' in the fourth column will let you see the files connected to the process, which can be useful if a process is spoofing its name or the file that was run was deleted by the process to try to cover its tracks. Entries with 'IPv4' in the fifth column will show any network connections that processes have opened up (visible on their own using 'lsof -i') in case the bot is trying to call home. Regards, Geoff
I don't mind at all! The more pointers the better! I'm new to VPSing and even newer to hack-hunting ... this is really useful.
Thanks again, Nick
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Nick,
most has already been said - and reinstall would be your best option anyways because you can´t trust the VPS anymore.
But one thing has not been mentioned yet I think:
Am 26.10.2014 um 19:53 schrieb Nick Sheppard: [...]
As for how it got in, most people seem to suspect an attack through ssh. But when I ssh'd in yesterday to start investigating, I was careful to note the "Last Login" message, and it gave my own genuine last login, three weeks ago from my home IP. Can I take this at face value to mean that ssh has not been used between my own two logins? Can "Last Login" be falsified?
It can be falsified. Last login IIRC is taken from wtmp. If the attacker gained root it would be easy to manipulate that too.
Cheers, Chris
tor-relays@lists.torproject.org
-
Dan Rogers -
Geoff Down -
Jeroen Massar -
Krbusek Christian -
Nick Sheppard -
Roger Dingledine