Log4j: Letting the JNDI out of the bottle

By |Published On: December 15th, 2021|Tags: , |

If you haven’t already seen it, we released a serious security advisory for Java applications using any version of the Log4j2 library less than or equal to version 2.14.1. How would you know whether or not your Java application is vulnerable? Well, chances are, if it uses Log4j2, accepts any kind of user-controlled input, and you haven’t already upgraded your application to use version 2.15+ of Log4j2, it’s very likely that it’s vulnerable. 

Before we dig in any deeper into this subject, I want to preface things by saying that I am by no means an expert on Java or Log4j2; I’m just dedicated to learning enough about subjects likely to be dangerous and/or impactful to my job. So, if there are mistakes or inaccuracies, I ask you to forgive me and perhaps expand my knowledge and the knowledge of others in the community.

Intro to Log4j2 Lookups

Do you want a relatively safe way to test whether or not your Java application is vulnerable without causing any major problems? Give your Java application the following input:

Copy to Clipboard

Let’s assume that the current date is December 13, 2021. If you input the string as directed and get the output 2021-12-13, then that means that the user was able to perform a lookup and inject data into the logs that Log4j2 writes for that application.

What, exactly, is a lookup? 

Lookups allow programmers to add values to the Log4j configuration in arbitrary places, according to the official documentation. If all this vulnerability did was allow users to print date strings and/or variables to the log, the impact would be minimal (provided any sensitive data stored in variables that were written to the logs weren’t able to be accessed by unauthorized users). 

So what makes this vulnerability such a big deal? To understand that, we need to talk about JNDI.

Letting the JNDI out of the bottle

The Java Naming and Directory Interface (JNDI) is an Application Programming Interface (API) that allows Java developers to provide naming and directory functions to Java applications. To put it another way, JNDI allows Java applications to use directory services like LDAP (think: Active Directory Lookups) or naming services like DNS, along with a handful of other methods. 

For our purposes, let’s focus on JNDI’s ability to do LDAP lookups, since that appears to be the method most are using to trigger the remote code execution portion of this vulnerability (in some cases, however, the DNS endpoint has been used in active exploitation attempts as well). 

Using JNDI’s LDAP or DNS options, you could then force a Java application to perform a DNS lookup. In some of the earliest instances of people testing this vulnerability, this was observed very frequently. For example, in the LunaSec blog, they linked to a Twitter post in which a user changed the name of their iPhone to the string ${jndi:ldap://[subdomain].dnslog.cn/a}, and not long after, they observed IP addresses from Apple’s IP address space attempting to resolve the subdomain they registered on dnslog.cn.

What’s really interesting is that the ability to perform DNS lookups in combination with the ability to pull environment variables leads to an unexpected attack vector: acquiring AWS variables and credentials. If you were an attacker, you could register a domain, set up an authoritative DNS server for that domain, log queries to the DNS server, and then perform Log4j2 exploits against cloud services:

Copy to Clipboard

Of course, that would return an NX domain, but a vulnerable server would query your DNS server, and that nonexistent domain, containing an AWS access key would be in the attacker’s DNS logs.

This alone probably would have been something significant to worry about, so what makes this vulnerability even worse? In addition to understanding JNDI, we also need to discuss Java’s “trustURLcodebase“.

Trust Issues

Without getting too deep into the weeds, Java has this feature called “trustURLcodebase“. Prior to JDK 6 update 211, JDK 7 update 201 JDK 8 update 191, and JDK 11.0.1, this feature was set to true. This feature would allow the Java application to remotely load any Java class file it was instructed to load. 

As mentioned above, JNDI allows lookups to be defined using a variety of different protocols. What makes the LDAP lookup work is that specific tools such as JNDI Exploit and marshalsec use a method called LDAPRefServer that redirects LDAP queries to where they can pick up malicious JAVA class files, such as an HTTP server.

Bringing it all together

In order to be able to trigger a remote code execution, the Java application has to meet a number of requirements:

  1. It needs to be running Log4j2 <=2.14.1, 
  2. It needs to log user input via its Log4j2 implementation, 
  3. It must be able to do JNDI – either LDAP or DNS, and
  4. It must be running on a version of Java with the trustURLcodebase parameter set to “True”.

If all these criteria are met, then it’s possible for attackers to trigger full remote code execution. Here is a diagram that demonstrates that that would look like:

In the diagram above, an attacker wants to attempt to exploit a Java web application. This hypothetical Java web application uses Log4j2 to log HTTP requests. The version of Log4j2 that implements logging for this application is vulnerable to the JNDI lookup vulnerability, and it is running a JDK version that has trustURLCodebase set to true.

The attacker issues an HTTP request in which they modified their user-agent string to issue a JNDI lookup:

Copy to Clipboard

This user-agent gets logged by the Java web application. Log4j2 sees the JNDI lookup and honors it. The web application server looks up the domain seemsleg.it, and issues an LDAP query to:

Copy to Clipboard

The LDAP server listening on port 1389 on seemsleg.it is designed to respond to the query and tell the client that the resource they’re looking for is located elsewhere. In this case, it tells the client who made the request to download the java class file at:

Copy to Clipboard

Since trustURLCodebase is set to true, the web application server sends an http request to seemsleg.it:8888/notmalware, downloads the malicious java class file, and loads it. 

Other Avenues of Exploitation

Let’s assume your software vendor decided to patch this vulnerability by blocking certain strings from being logged, such as:

Copy to Clipboard

There are bypasses to get past brittle blocklists that rely on detecting these strings. For example:

Copy to Clipboard

These two strings would end up normalizing to:

Copy to Clipboard

The Sophos blog recently included a post with even more blocklist bypasses that have been observed in the wild. In addition to blocklist bypasses, it’s been suggested that there are other JNDI methods that could be used to exploit vulnerable servers. The LunaSec blog post mentioned earlier links to the Veracode blog post that shows dedicated attackers could try to leverage existing code on a server (sometimes referred to as a “gadget“) to gain code execution.

Possible Attack Surface

Y’know whenever you go to install the latest JRE update, and while you’re in the middle of installing it during the load screen the little banner that pops up saying “Billions of devices run Java”? Well, that’s not just marketing fluff. There are tons of devices out there running some form of the Java runtime environment. 

While every device out there running Java might not be utilizing the Log4j2 library, know that there are enough applications out there that are that this vulnerability will likely be a problem for months if not years to come as companies, organizations, and vendors scramble to figure out which Java applications of theirs use Log4j2 and are therefore potentially vulnerable–and the attack surface is pretty wide.

As an example of an unexpected attack surface, on Friday I saw security researcher John Hammond tweeting about getting remote code execution in the popular Java-based video game Minecraft. That inspired me to try it on my own. After a day of futzing around, I managed to exploit Minecraft, but with some caveats. If you’re curious, see my video about it below: 

The associated presentation–including clickable resource links–is also available for download here.

Here are some of the vendors who have confirmed they were using the vulnerable Log4j2 libraries and their recommendations going forward:

As time goes on, it is highly likely that other IT and IT security vendors will post their own advisories related to this vulnerability. I came across this GitHub repository of vendors who have released advisories and guidance for their products in addition to the links above. Additionally, CISA, in collaboration with security researcher Kevin Beaumont, made a spreadsheet of affected products, and they provided links to security advisories from the vendors for those seeking further guidance.

The worst part about all of this is that there are likely software stacks, appliances, and applications that will never be upgraded. Either due to the software/appliance being considered end of life, the software project being considered too important and business critical to update, and/or cases in which either the company that wrote the software or the developers are no longer available to update the application. 

Mitigations and Remediation

As usual, the most obvious path for remediating this vulnerability is to update to the latest implementation of Log4j2. As of right now that’s version 2.16, in which they’ve disabled the lookup functionality in its entirety. 

Of course, because this is a library that is likely to be a part of a much larger Java application or applications in your environment–some of which you may have the ability to modify and many, many others that you will likely have to wait for vendor patches to actually do anything–patching it by updating the vulnerable library is going to be much more easily said than done.

If you’re using Log4j2 between versions 2.10 and 2.14.1, set this option on the command line when running your Java application:

Copy to Clipboard

Alternatively, you can also set the environment variable:

Copy to Clipboard

to achieve the same effect.

Recall that in order to have the vulnerable application download and run the malicious Java class file, they need to be running a vulnerable version of the JDK with the trustURLcodebase parameter set to true. 

If you are able to update the JDK version being used to run your Java applications, then that would serve as a somewhat effective mitigation against the LDAPRefServer exploitation method. Recall that other exploitation methods (exfiltration of environment variables and use of existing gadgets) still exist, and they could still be used to target vulnerable Java applications.

Now, what if you don’t have the ability to update the library, change the command line arguments, or change the version of the JDK your Java application uses? 

Well, if you’re feeling particularly brave–and you’re absolutely sure that your organization doesn’t use any of the JNDI lookup functions in the log4j2 library–you can use the zip command to delete them out of the log4j2 library’s JAR files:

Copy to Clipboard

Considering the fact that you’re ripping out entire java class files from the library, I would highly recommend testing this mitigation before messing with any production applications.

What if you don’t have the option to use any of these mitigation methods? 

Consider the fact that this exploit requires the system being attacked to make DNS, LDAP, and/or HTTP connections outbound. Here are some questions related to defense in depth and security monitoring to consider: 

  • Does the server have a firewall deployed? Is there an egress filtering policy applied? 
  • Is there a firewall at the perimeter of your network or your cloud deployment? Is egress filtering being applied there? 
  • Is there an HTTP proxy that your systems are required to use for HTTP connectivity? 
  • Do you have DNS query logs available? Do your logs identify source IP addresses that are making the DNS requests? If so, consider searching for DNS queries where the source IP addresses are your java application servers and/or appliances, and sorting them by least frequent occurrence.

These are all effective network boundary defenses that can stop the exploit from succeeding, as well as invaluable data sources for threat hunting in order to search for anomalies and/or possible exploitation attempts.

In cases where network boundaries cannot be deployed or there is insufficient logging, consider endpoint protection solutions to detect post-exploitation activity, such as: 

  • EDR/XDR solutions
  • sysmon
  • osquery
  • auditd

If the Java process for your application is suddenly making outbound connections it doesn’t typically make, starts running child processes it doesn’t typically spawn, or drops new files to disk that have never been seen in the network before, know that these are anomalies worth investigating. Hopefully your EDR/XDR tools will make that task easier.

Conclusion

I’ve heard security researchers stating that this “Log4Shell” vulnerability has a “long tail.” What that means is that since Log4j2 is so ubiquitous–and because it is the de facto standard for logging anything in java applications–it’s highly likely that we’ll be seeing a lot of potential attack surface with this vulnerability, some of it obvious, some of it not so obvious. 

It’s also likely that due to the nature of this vulnerability and how java applications are the backbone of organizations and entire business verticals all over the world, patching this vulnerability will be no easy matter.

The best advice I can offer is to do the best you can with what you have. Take it one step at a time. I’ve often been told that cybersecurity isn’t a sprint, but a marathon. Make sure you take care of yourself, your colleagues, and your peers. Remember to step away from the keyboard once in a while, and–especially with the holiday season coming up–take time for yourself, your family, and your friends.

Happy holidays, and happy hunting.

Share with your network!
Get monthly updates from Hurricane Labs
* indicates required

About Hurricane Labs

Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.

For more information, visit www.hurricanelabs.com and follow us on Twitter @hurricanelabs.