log4j Casualty - The Unifi Controller

With all the publicity given to the log4j vulnerability, customers nervously asking if their servers and applications are vulnerable, where your answer is always a categorical 'no' (sometimes accompanied with a rationale that the vulnerability stems from hosting a Java application that makes use of the log4j library and is exposed to the Internet), it turns out you have overlooked your own backyard for that one application fulfilling all three conditions above. And not just that, but the first time in a year or so, you fell behind with updating it to the latest version.

Problem detection: Monitoring says "Something's wrong"

An alert triggered by the monitoring system reports high RAM usage (surpassing the critical threshold) on the Unifi controller server.

This is rather odd, but the sensor monitoring the web UI port remains silent - the service is up and running. So, no biggie, we'll look into it in a bit, right? Wrong!
Closer inspection of the server suggests there's more to it:

• The CPU is maxed out on all cores
• The process consuming all the CPU cycles, "sidekiq", is running as the unifi user
• The process seems alien, but you Google "sidekiq+unifi" just the same, doubting yourself and suspecting it's the Unifi controller that's gone haywire. But, you come up empty-handed.
• Restarting the Unifi service exits cleanly, but the "sidekiq" process remains unaffected

Then, when a 'java' process catches your eye in the process list, you realize that the Unifi controller service is running inside Java VM and remember that the service is facing the Internet... Okay, there's something to worry about.

First, you check for available Unifi controller updates and confirm there is one, patching log4j and addressing CVE-2021-45046 vulnerability.

One could argue Ubiquity has downplayed the importance of it, marking the fix as "Improvement"
(see https://community.ui.com/releases/UniFi-Network-Application-6-5-55/48c64137-4a4a-41f7-b7e4-3bee505ae16e)
and not immediately raising a red flag with you - in spite of the bad gut feeling.

Terminating the "sidekiq" process hogging the CPU and confirming that the system is compromised

• A simple TERM signal was sufficient to kill the "sidekiq" process, and as it turns out - there was no parent process to respawn it
• Because an alert was triggered less than 24 hours ago, running the "find" command discovered some very recently modified files sticking out by their naming convention and/or by the file system location:

# find / -type f -mtime -1 | egrep -v '^/proc|^/sys|^/run'

/var/lib/unifi/.bash_history
/tmp/.jeeej/c.j
/tmp/.jeeej/sidekiq

From this point on, there was no doubt the system is compromised and you have to:

• Assess the damage and take necessary steps
• Patch the vulnerability

The fix

The immediate course of action taken was to:

• Take a snapshot of the compromised server for further assessment of what the impact is and could be
• Restore the most recent snapshot of this server from the backup and update the Unifi controller (better safe than sorry, right?). Thus, you restore a clean snapshot of the server and don't have to worry if the malware cleanup process was comprehensive. Also, this quickly restores the Unifi controller service
• Conduct forensic analysis on the compromised server snapshot image

There's also the matter of a legacy Unifi controller managing legacy devices. This piece of software is not receiving updates for years now, so, two options remain to patch this system:

• Restrict access to a trusted handful of IPs and subnets
• Try to get hold of patched log4j JARs compatible with this Unifi controller version and deploy them in place of the existing JAR files, fol. by the Unifi controller service restart

Post-mortem

Analyzing the snapshot of the compromised server snapshot, binaries, and other extraneous files found that the ultimate goal of the intrusion was to mine Monero cryptocurrency

• The vulnerable Unifi controller web UI (which ships their own log4j library) was exploited (I have no proof of this in the form of web server logs, but considering the intrusion was used for RCE as the unifi user, it seems pretty obvious)
• The RCE invoked curl/wget download (from the Oracle cloud) the 'cc' Bash script and pipe it for execution (the sole command listed in the /var/lib/unifi/.bash_history file)
• The script cleans up everything from /tmp/ and downloads a tar.gz archive containing the Monero mining binary and the configuration file (this is downloaded from Singapore IP address, but it's probably accessible binary/config. to anyone who wants to mine Monero)
• The 'cc' script then updates the Monero mining configuration file with the Monero "User id", renames file/folder names from the extracted archive, and runs the binary used to mine

The script doesn't even bother to clean up behind it (the /var/lib/unifi/.bash_history, for example), so I believe this is as far as this compromise goes and we got off easy.

Lessons learned

As technology advances, it's becoming increasingly complex and hard to keep up with. System security is a moving target that needs a proactive and layered approach.

That being said, and although we got off easy and this was more of a wake-up call, leaving the application unpatched was still a lesser blunder than not having a valid backup, or a proper monitoring system configuration to alert you would have been.

While it may seem useless, a monitoring system alerting you of anomalies in network throughput and CPU/HDD/RAM usage can prove invaluable.