Inspecting and patching your *nix system for security vulnerability CVE-2021-44228 AKA Log4Shell

Log4Shell under Linux

What is Log4Shell?

Log4j is a widely used Java-based library. It is a piece of ready-made code, that could be used by developers to integrate logging abilities in their software. It has been favoured over the in-built System.out.println function for a good while now because it allows for extensive logging abilities and configurations.

Security experts discovered a severe exploit in the Log4j library code which can allow attackers to remotely execute malicious code on targeted systems, leading to the attacker gaining shell access to the system – hence the name.

How severe is the Log4Shell vulnerability?

Extreme. On the CVSS scale it has a full score of 10.0. This as high as it gets and it basically means that attackers who can leverage this exploit against one of your systems can gain full (root) control over the system. You must inspect, update and/or patch any affected systems immediately.

The Log4Shell review on the CVSS website.

Is it easy to detect and fix the Log4Shell vulnerability?

Looking at the way Java applications are composed it can be quite challenging to detect and patch the Log4j vulnerability because Java applications are distributed as JAR archives. Those archives can contain multiple, smaller, JAR archives (called dependencies) which they need during run-time. Any of those “nested” JARs could contain a version of the affected Log4j library. It is not an infection or a piece of malicious software. It is a legitimate, and quite useful, Java library and the removal of this library can cause your software/system to malfunction.
Another aspect is the popularity of Log4j. Almost every Java application out there logs its run-time data, and nothing makes this job easier than the Log4j. The Log4j could be lurking within your system without you knowing.

Which systems are affected by this vulnerability?

Generally, systems running Apache web server and Java binaries are most vulnerable. If your system has either or both of those you should be very concerned. See below for tips on how to check for and rid your system of the affected pieces of software.

How can I check if my system is affected?
How can I scan my system for the Log4Shell?

First, you need to answer two simple questions:
  1. Is your system running Apache web server?
  2. Does your system have the Java binaries or any Java-based applications installed?

If the answer to both questions is a solid No, then you should not worry – but you are encouraged to run the scan anyways. I think it is a good exercise on system maintenance.

Second, you can use any or all of the methods below to inspect your system:
1. Check if you have Java and/or Apache installed installed

$ java -version
$ apache2 -v
$ dpkg –get-selections | grep apache*

If those commands return a positive result, continue below…

2. Manually scan for the Log4j class. Execute the following command as root on the system

Using the “find” command can quickly point out the path to the Log4j library. This command can be used on both Linux and FreeBSD systems.

$ find / -name “log4j*”

Search the entire file system for Log4j traces using the find command .

3. Find all Log4j references in the configuration files.

This can help you if, for instance, you cannot immediately update your system and want to comment out the call for Log4j

$ find /etc/ -type f -name “log4j*” -exec grep –color -Hni “log4j” {} ;

An affected system running Jitsi’s java-based videobridge software.

4. Use one of the available scan scripts

Some developers have already written simple scripts that you can use to scan your system. One of the available scripts is this one on Github, by Rubo77.

$ wget https://raw.githubusercontent.com/rubo77/log4j_checker_beta/main/log4j_checker_beta.sh -q -O – |bash

Log4Shell in detail
Using the Log4j checker script on an Ubuntu-based system. The script was able to point out the affected packages.

5. Use a dedicated open source scan & patch tool designed for the Log4shell

There are Log4j-specific scan tools available. One of those is the Log4j2-scanner utility from Logpresso. This utility scans for vulnerable code and offers you the ability to “patch” affected pieces of software. Before patching, the tool creates a backup of the files it will modify. The file will have the .bak extension. This is quite useful because you can restore those backups in case your system malfunctions or exhibits instabilities post-patch.

Here you will learn how to use the Log4j2 scanner:

$ wget https://github.com/logpresso/CVE-2021-44228-Scanner/releases/download/v1.3.2/logpresso-log4j2-scan-1.3.2-linux.tar.gz
$ tar -xvzf logpresso-log4j2-scan-1.3.2-linux.tar.gz
$ chmod u+x ./log4j2-scan
$ ./log4j2-scan / (this is a scan-only mode)
$ ./log4j2-scan –fix / (this is the scan and fix mode, you need to confirm this action)

Log4Shell in detail
The Log4j-scanner tool from Logpresso showing the two possible use modes and the backup file it creates.

6. Use an open source vulnerability scanning tool

There are plenty of free and open vlunerabitity scan tools that can help you diagnose your system. Personally, I favourite Grype and Syft. Here is how to use Grype to scan your system… Here you can learn how to scan your system with Grype…

The latest version of Grype can be found here. For other versions, adjust your commands accordingly

$ wget https://github.com/anchore/grype/releases/download/v0.27.2/grype_0.27.2_linux_amd64.deb
$ dpkg -i grype_0.27.2_linux_amd64.deb
$ grype dir:/

Log4Shell Log4j
Grype scan result on an Ubuntu-based system. System’s clean.

Log4Shell Log4j
Grype scan result on another system. The results indicate vulnerable packages and their level of severity.

Fixing the Log4j issue in NAKIVO Backup & Replication for Linux

Users must remove the JndiLookup.class file from the log4j-core-2.2.jar jar file using the following steps:

  1. Using a terminal shell, navigate to the “libs” folder located in the NAKIVO Backup & Replication installation folder.
  2. Execute the following command:
    $ zip q -d log4j-core*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
  3. Restart NAKIVO Backup & Replication.

According to NAKIVO, the software should then work again as expected.

Fixing the Log4j issue in NAKIVO Backup & Replication for FreeNAS/TrueNAS

On the FreeNAS/TrueNAS systems, the NAKIVO software runs inside a Jail. You will need to login into that jail and install the zip binary to be able to fix the issue. Here is how:

  1. Login into your FreeNAS/TrueNAS GUI, scroll down and, on the left side, click on the “Shell” button.
  2. Type the following command to login into the NAKIVO jail

    $ iocage console nbr
  3. Install the zip binary

    $ pkg install zip (you have to confirm this step with Y)
  4. Navigate to the installation directory and patch the affected JAR

    $ cd /usr/local/nakivo/director/
    $ zip -q -d log4j-core*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

According to NAKIVO, the software should continue to function as expected afterwards.

Examining your NASDeluxe system for Log4j traces

NDL2xxx
The screenshot was taken from a NASdeluxe NDL-2880R with firmware version 2.06.03.

1. Login into the GUI and enable the SSH service under Network Service > SSH
2. Use a terminal/SSH client to connect as root to your NASdeluxe system IP. The password should be “admin” unless you have changed it.

$ ssh root@NASDeluxe_IP

3. Run the following commands to verify that your system does not contain the affected library or the Java binary:

$ find / -name “log4j*”

$ find / -type f -name “log4j*” -exec grep –color -Hni “log4j” {} \;

$ java –version

IMPORTANT: The first two commands should retun an empty prompt while the third command should return an error “java: command not found

4. To manually inspect what libraries are included with the installed Apache verver you can list the contents of /opt/apache/lib/ directory like so;

$ ls -lah /opt/apache/lib/

This should visually confirm the abcense of the lib4j library.

5. Exit the SSH shell by pressing CTRL + D or typing “exit” and pressing Enter.

6. Go back to the web GUI and disable the SSH service.
Please check our relevant technical announcement for more details and tips on inspecting and securing your systems

Updating your Linux-based system

Software providers have been racing to release security updates for this vulnerability. It is possible an update has been already released for your system. To check and install new updates on your Linux-based system you can execute the following command:

$ apt update && apt list –upgradable

This will list all possible updates. You can carry out all available updates with the command:

$ apt upgrade -y

If you wish to upgrade only the affected packages you can use the command:

$ apt –only-upgrade install package_name

Log4Shell in detail

Important note

Apache.org released an early patch to fix the Log4Shell (version Apache Log4j 2.15.0) which was proved to be incomplete and is still vulnerable under specific running configurations. If you applied the above patch your system is still exposed. You must update once more to the latest secure version. Please check this Apache.org page for more information.

Are there any other steps we could take to further secure our systems?

Yes, there are some things you could do to improve the security of your system. Here are some ideas: Here are some ideas:

1. Use iptables rules to restrict connections to known safe IPs

This is a remote code execution (RCE) exploit. It relies on the compromised system being able to fetch malicious code from a remote server, which the attacker controls. Restricting communication to a list of Trusted IPs can stop this exploit in tracks. You can do so by adding a few rules to your iptables. If your application is running on a Linux machine you could run the following commands:

Start by saving your current iptables rule set:

Debian/Ubuntu

$ iptables-save > /etc/iptables/rules.v4 (this stores your current iptables IPv4 ruleset)
$ ip6tables-save > /etc/iptables/rules.v6 (this saves your current iptables IPv6 ruleset)

CentOS/Red Hat/Fedora

$ iptables-store > /etc/sysconfig/iptables
$ ip6tables-save > /etc/sysconfig/ip6tables

Important: If you are managing this server remotely you must allow SSH connections first. Otherwise you might get blocked from the server. Assuming you are using the defualt SSH port, do;

$ iptables -A INPUT -p tcp –dport 22 -j ACCEPT

Now you can start adding rules:

$ iptables -A INPUT -i lo -j ACCEPT
$ iptables -A INPUT -m state –state RELATED,ESTABLISHED -j ACCEPT
$ iptables -A INPUT -s 10.0.0.XXX/24 -j ACCEPT (Repeat this step for all your Trusted IPs. Edit IP/Subnet)
$ iptables -A INPUT -m state –state INVALID -j DROP
$ iptables -P FORWARD DROP
$ iptables -P INPUT DROP

What you are doing here is telling your system’s built-in firewall to: accept packets from the local loop address, accept valid and relevant packets, accept packets from a specific IP, stop forwarding packets to other systems within and outside your network, and to drop everything else.

If you apply the above rules please monitor your system/application to make sure it is interacting as intended with other systems it should connect to or accept connections from. Modify the rules accordingly. If you have to restore your previous rule set do;

Debian/Ubuntu

$ iptables /sbin/iptables-restore < /etc/iptables/rules.v4
$ ip6tables-restore < /etc/iptables/rules.v6

CentOS/Red Hat/Fedora

$ iptables-restore < /etc/sysconfig/iptables
$ ip6tables-restore < /etc/sysconfig/ip6tables

Restrict access to required ports only

Again, the exploit needs to communicate over a unique open port. If you shut off all ports, except those strictly required by your server, you have a better chance to avoid being compromised. Assuming you system needs only ports 80/TCP and 443/TCP to function you can do the following:

$ -A INPUT -p tcp -m tcp –dport 80 -j ACCEPT
$ -A INPUT -p tcp -m tcp –dport 443 -j ACCEPT

3. Secure your JVM

The exploit also relies on relaxed configuration of your Java environment. You can modify the Java property responsible for log4j’s JNDI URL look ups. Setting the property to “false” is recommended here.

com.sun.jndi.ldap.object.trustURLCodebase = false

You should also set the configuration value log4j2.formatMsgNoLookups to true. This will disable the LDAP queries.

4. Use NGINX with ModSecurity WAF and LUA scripts

This is a more radical and involving solution. But security and performance gains might actually be worth it on the long run.

NGINXis a popular Open Source, fast and lightweight web server. NGINX is not vulnerable to the Log4Shell exploit because it is written fully in C language, does not use Java, or any Java‑based libraries. ModSecurity is an Open Source Web Application Firewall (WAF) module available for NGINX and other web servers. It uses a set of generic attack detectionrules called Core Rule Set (CSR) to protect web applications from a wide range of attacks, with a minimum of false alerts.

There are also some useful LUA scripts that can improve NGINX’s resistance to such attacks. LUA scripts require the NGINX module.The good guys at Infiniroot hosting have published their useful LUA scripts on this Github page accompanied by this excellent blog post.

Transition to NGINX might take you a while though because site configurations are not corss-compatible between Apache and NGINX. It might be useful to first experiment with a local lab server to make sure your site/application will run smoothly with NGINX.

Starline contact

Any questions? Please contact us.

This proven expert for Linux and Ceph platforms joined Starline in 2018. His preference is for sophisticated open source solutions and tricky product developments. Therefore he is also available for inquiries regarding PetaSAN and FreeNAS. But he also likes ARM servers from Ambedded or Mac operating systems. In his private life, the engineer enjoys working with 3D printers and robots.

Mohammad Ammar