JMX — The Best Buddy of EVERY Red Teamer.
Have you ever been (as a red teamer) set under a very high pressure when it comes to a successful outcome delivery from the ongoing operation? If you are in the business at least for a couple of years already, then it has for sure happened not once or twice and you will also face it very soon in the future. Imagine you have achieved a successful foothold in the targeted internal corporate network of your client. Perhaps a careless employee opened an attachment from your phishing campaign, you got lucky connecting unauthorized, remotely controlled device or just in any other possible way. You perform a slow, cautious reconnaissance trying to find any useful information, like valid credentials over-left by IT/DevOps staff or business employees in organization repositories, any technical documentation leading to exploitable vulnerabilities or just by passively analyzing network traffic. Time is ticking away and the list of your available options becomes shorter with every consumed day. What would you do in such situation? Would you target organization’s Active Directory services following the classical red teaming cookbook and risking detection by the client’s Cyber Defense Center (CDC) or rather go hunting for Remote Code Executions (RCEs) in the corporate environment and possible lateral movement opportunities? Over 95% of professional red teamers would chose the first option, I would however rather go hunting.
Java Management Extensions (JMX) — Absolute Top of the Black-Box Hacking Silver Bullet List
A successful black-box hunting for RCEs requires having a list of promising targets, vulnerabilities commonly present in an average business environment, a list of “the best red teamer’s companions” who never disappoints and always offer helping hand when the latest is needed the most. I have developed my own hit list over the years of my professional practice and misconfigured JMX ([1]) holds a very strong position in my personal corporate hacking TOP 5. Why is it so attractive for a potential adversary and what are the main benefits from the offensive viewpoint perspective? I can see a number of them, in particular:
- Java (JVM) is still the dominant technology when it comes to development of applications, APIs, Web Services and other solutions consumed by commercial, public and non-profit organizations. Even considering the fact that other technologies have expanded their overall market share in the last couple of years, Java is still the number one.
- JMX is a very popular technology for managing and monitoring applications, system objects, devices (such as printers) and service-oriented networks. In a big corporate environment with dozens of thousands virtual machines, the number of listening network interfaces can easily exceed many thousands.
- If only a fraction of active JMX interfaces is misconfigured, a simple statistical calculation leads us to the conclusion, that number of exploitable RCEs can be even higher than a couple of hundreds in the aforementioned environments.
- Misconfigured JMX can be easily and safely exploited during Red Teaming operations. Reliability of the Java technology and JMX exploitation has been proven over the last decade and the White Team participating in the operation will not have any problems in giving green light for exploitation of the identified vulnerable instances.
- Last but not least — Java is present and whitelisted on nearly all corporate user endpoints over the World. Big organizations cannot now nor in any near future remove this software because of many compliance requirements of legacy applications.
Exploitation of JMX enjoys a limited popularity (so far) among professional red teamers mostly due to the fact that its network interface does not have assigned any standard, fixed TCP port number. Later in this article I will prove that this is not only a half-truth but also demonstrate how to overcome potential difficulties in vulnerable instances discovery. From the other hand also real active threat actors still mostly focus on the most popular CVEs and corresponding, publicly available exploits. Misconfigurations exposing RCE conditions remain less popular, but it can be a subject to change in a near future, so organizations should prepare themselves for that.
We will focus here on the most common JMX exploitation and achieving RCEs by abusing management applets (MLETs). This technique is already quite well documented in many technical articles (see [1]) so the main purpose of this article is an attempt of solving various practical challenges in JMX exploitation during red teaming operations rather than inventing the wheel again.
Technical Documentation — Where the Whole Evil Gets Born (Red Teaming OSiNT).
Let’s start our JMX Open Source Intelligence (OSiNT) journey. Unfortunately, the best practices in securing exposed JMX network interfaces have been constantly neglected over more than the last decade. Let’s consider official, recent JVM (Java Virtual Machine, or just Java) Oracle documentation in Figure 1.
We can see that the vendor indeed warns from using described insecure Java configuration (Disabling JMX Security) and informs about the risk of making Java application vulnerable to arbitrary code execution of intruders’ choice. From one hand the vendor warns from the insecure setup in production environment, while from the other hand suggests that it might be acceptable for development. Oooh is it?! Really??? While separation of the development (and testing/acceptance) environments from the production ones is considered a common best security practice, various types deviations from this policy is our everyday corporate reality. Compromised test and development environments are by most of experienced red teamers treated just as a stepping stones towards the production, nothing more.
Digging even Deeper — Just Give me all the TCP port numbers used by (insecure) JMX instances to listen!
Official Oracle Java SE documentation is just a humble beginning of our OSiNT journey. There is of course way more to explore and collect from a malicious adversary or an ethical hacker’s viewpoint. Let’s take a look at Figure 2 presenting configuration of the Network Derby Server.
This documentation is (as a matter of fact) quite old and pertains JDK 6, but still available on the official vendor’s web site for the software users. Insecure setup and disabled security features are mentioned, but without any clear indication that such configuration is too dangerous to have any real practical use. Please also note the TCP port number (9999) and refer to my other article [3] from October 2022 describing technical details of critical RCE (CVE-2022–24082) discovered by me in the PEGA Platform. Wasn’t the vulnerable application listening on exactly the same port number? A coincidence? I honestly dare doubting, but even if so in this particular case, the number 9999 is worth to note for later use in red teaming operations. Especially taking into account the content of another, newer comparing to the Derby Server above Oracle’s tutorial, describing how to expose resources for remote management by JConsole — see Figure 3.
Instead a clear, distinct warning/caution note we have here a “Security consideration:” section stating that authentication and encryption mechanisms are disabled in this particular example and should be implemented in real-world (what does it mean exactly?!) environments. The message is even softer than the one from Java SE 9 documentation and again TCP port 9999 is in use (this number is really worth to note and remember!). Now let’s examine two other examples. Figure 4 shows another piece of Java documentation, this time SE 8 (v5 is also explicitly mentioned in the content).
Although the documentation refers “Local JMX Connections”, given setup can easily become a remote target as well. Moreover, this time no single warning or note is present about running possible security risks and no advisory at all about making the whole setup secure. From the hacking perspective two more TCP target ports to record: 3333 but also 2222 found in the later part of the documentation article, not shown in Figure 4 above.
In order to prevent getting impression that Oracle is the only “bad” boy in the room, let’s review another example presented in Figure 5. This time our subject of examination is an IBM product called Engineering Lifecycle Management (ELM, [4]). Its documentation describes insecure, unauthenticated configuration of JMX for Apache Tomcat supporting this software. Important exclamation mark section (!) states that presented setup allows connecting the interface without submitting credentials and is not recommended for production environments. Unfortunately it is not even mentioned that the whole configuration is insecure and very dangerous for its users. Moreover, the documentation states that for experimenting in local and test environments it is just fine (useful)! The corporate Cybersecurity reality is so badly ignored again! From the intruder’s perspective we have another TCP port number for our list, this time 1099. This port requires a special attention because as shown in Figure 6, it is the default port for RMI (Remote Method Invocation, see [5]) registry and as such can also be considered as the default JMX TCP port. In a large enough business environment it will pop up for sure during the active reconnaissance phase of your red teaming operation at least several times.
Show Me Your Hand! — Confronting Corporate Reality.
The above described, short OSiNT reconnaissance gives us a nice outcome with four TCP ports (9999, 3333, 2222 & 1099) for our JMX hunting in the client’s internal network. Consider it however only as a humble starting point. A more extensive research would deliver a way longer list and I’m leaving this exercise as a homework for the readers. The TCP port list can also be enriched by combining the numbers collected during OSiNT with some extra additions (e.g. 11099, 10099, 19999, 33333, 22222 etc), because IT/DevOps staff likes to include such additions in their configurations as well. The aforementioned list cannot be also too long, because from one hand red teaming operations are always time-boxed and from the other hand intensive scanning can be easily detected by the CDC. Nevertheless with our outcome from OSiNT we can start active reconnaissance. I will skip the network (TCP/IP) part, because describing hiding (from CDC) techniques exploited in red teaming operations deserves a separate article and perhaps I will come up with one soon in the future. In the meantime let’s focus on JMX itself. Following a regular pentesting practice we would use nmap network scanner (see [7]) with the option -sV to identify RMI interface and the rmi-dumpregistry NSE script to receive RMI registry content. Use of nmap is in red teaming however very risky, CDC detection tooling has all sort of nmap-aware signatures included and reacts highly allergic to any scan attempts, even the slow ones. Instead it is way safer to write own, dedicated tool in Java. Due to historical compliance reasons, JVM is still by default installed on most of the corporate laptops over the World so you will rather not face problems with execution of your code. On top of that Java is also practically always white-listed in AV/EDR setups so you can consider this as a double win. To make our JMX “service scanning” as efficient as possible, we can just attempt to invoke and non-existing method (from an non-existing class) over the open ports identified during the network scanning. Such invocation can result in one of the three possible outcomes:
- The tested port does not serve JMX RMI. In such case our connection will be terminated at some point with (or without) an unidentified output.
- The tested port does serve JMX RMI, but it is secured with username/password authentication and/or TLS. In such situation our connection will be terminated with a known, familiar output.
- The tested port does serve JMX RMI and is setup with disabled security (vulnerable). In such situation our connection will be terminated with a Java exception in the output (Figure 7).
Security Warning!
Scanning described above performed on unidentified network services can in some rare circumstances give unexpected outcome. For certain types of services (e.g. legacy devices in OT networks or SCADA) there is a low risk of damage introduction. As such, ensure that the White Team participating in the operation is well informed and ALWAYS gives officially upfront green light for all your activities.
Overcoming Obstacles part I — How NOT to Get Deceived by Appearances?
One of the properties supported by Java for its RMI is java.rmi.server.hostname ([9]). According to the Java SE 8 documentation ([9]) “the value of this property represents the host name string that should be associated with remote stubs for locally created remote objects, in order to allow clients to invoke methods on the remote object. The default value of this property is the IP address of the local host, in “dotted-quad” format”. In other words the value of this property is the endpoint (IP:Port) exposed by RMI registry for connecting JMX clients. The problem is, that setting this property does not limit Java to this one single IP for remote methods invocations. All other network interfaces keep listening as well. In general it leads to the two following very common misconfigurations, notoriously present in typical corporate environments:
- Setting its value to localhost:port/127.0.0.1:port in a failed attempt to limit JMX exposure to the local loopback interface only.
- Setting its value to an IP address just different than the one assigned to the JVM host. Such situation can occur for instance when JMX was used earlier in a development or test environment, over left in the production but is not anymore really needed for any legitimate purpose.
Practical exploitation of the first from the above scenarios is described in [3] for my CVE 2022–24082 discovered in the PEGA Platform using MogwaiLabs JMX Exploitation Toolkit (see [8]). The second scenario is not very well publicly known, but is very common in typical business environments. Imagine VM with assigned IP address 192.168.229.131 running JVM from Figure 8.
Exploitation from other remote machine (with IP 192.168.229.128 in our case) would be rather simple in situation when we have root/administrator privileges on our attacker host. We just need to assign additional IP address 10.0.0.1 for ourself (interface alias in linux case works just fine) and redirect traffic coming to the port TCP/9999 on that address using socat tool:
The next step is dumping RMI registry from 10.0.0.1:9999 to see on which dynamic TCP port we can remotely invoke methods. In our case:
Now we need to run our socat again (to redirect TCP/41799) and we will be ready for the actual exploitation. We can use exploit of our choice and enjoy achieved RCE. An example with earlier already mentioned MOGWAI exploitation toolkit is presented in Figure 10.
What shall we however do, when we do not have root/admin privileges on our attacker host and we cannot setup any additional IP address? Such conditions are very common for instance when a foothold is achieved on a corporate laptop as a result of a successful phishing campaign. This problem is not really trivial to solve (but still doable) due to JVM peculiarities and I’m going to dedicate a separate blogpost for this subject in a near future.
Overcoming Obstacles part II — How to Deliver Our Payload?
Although it was not explicitly explained in previous section, our JMX exploit downloaded its payload from specially started http server (http://192.168.229.128:6666 in Figure 11). In practice the following getMBeansFromURL is called by the targeted JMX instance to upload the payload:
What is the real problem here? Now, during red teaming operations we are not always in position to get reverse connections from targeted JMX servers to hosts being in our control. A good example would be again corporate endpoints. In mature environments they are located in dedicated segments, where incoming connections are prohibited or strictly controlled. It is also not always easy (at least not during initial phases of performed operations) to find a suitable HTTP(S) Server spot in the corporate network, where on one hand we could upload our payload and from the other hand the targeted JMX server can download it. So what can we do? Let’s take a look at the getMBeansFromURL method specification again. It states “The URL of the text file to be loaded …”. Wait a minute …, is URL 100% equivalent to HTTP(S)? Not really, because we have a number of other possible protocols!!! I know where are you up to at this point, but I have to disappoint you a bit. SMB protocol did not work in my tests because smb:// was not supported by the discussed Java method. FTP however, works perfect! And this can be often a real game changer, because although the protocol is insecure by itself, it is still (unfortunately) widely used internally by most of the organizations. If we can only find an FTP server where we can write and from where our targeted JMX server can download then … we are golden!!! But first things first — lets specify, how our FTP URL should look like. Providing that targeted JMX server 192.168.229.131 would download the payload from the FTP server 192.168.229.128, we have:
ftp://user_id:user_password@192.168.229.128
The getMBeansFromURL method specification also states that “the text file to be loaded”. Let’s take a shortcut and look into the source code of mjet.py [8] to determine, how this text file should look like. In our case after some minor modifications it would be:
We can now save the text from Figure 13 in the file mlet.html in the default directory of our FTP server. The last missing piece is the file payload.jar. For testing purposes we can use MogwaiLabsMJET-MLet.jar from the payloads directory in [8], rename it to payload.jar and place in the default directory of our FTP Server as well. Having FTP Server started and all above arranged, we are ready to execute our exploit (Figure 14).
And voila, full success!!! Although as we can see in Figure 14, web server hosting the payload is still started on port TCP/6666, it is completely ignored during exploitation and our payload is downloaded via FTP.
Conclusions & Recommendations — I Will Fear no Evil!!!
Having all the knowledge about risks and danger coming from JMX misconfigurations it is time to ask yourself the question “What a modern organization can do to protect their assets from threat actors abusing this type of vulnerability?”. I can define at least several points of consideration:
- First of all you need to realize that Java is a technology which you will just not be able to avoid for many years forward. Do not try to convince yourself that this is only a temporary or tactical solution for you. And it does mean that you will live with JMX for many years as well. You have to define a secure standard of its configuration & use and ensure its organization wide enforcement. There is no other correct way and no other shortcut.
- When you already have the aforementioned standard defined, you need to find an effective way to enforce it in your environment. It means that all your IT/DevOps/CyberSecurity staff including contractors and other externals need to be trained and aware. They have to understand the risk and danger and have to acquire enough technical knowledge to apply necessary controls.
- How effective is your Vulnerability Management (VM) System? Can it correctly identify all vulnerable JMX conditions? Have you tested and checked it by yourself or you’ve just taken promises of your VM provider at face value? How fast end effective is response to newly identified vulnerabilities?
- And last but not least — How effective are your detection controls when it comes to attempts of JMX abuse (e.g. by invoking “infamous” getMBeansFromURL method)? Have you tested those controls by yourself or you again just believe your vendor? How quick and effective are your response capabilities?
Secure business environment and (self)confidence can only come by fully addressing all above consideration points and questions.
As the final conclusion I would like to ask you: “Do you agree that JMX is so far the best Red Teamer’s buddy?”. Even if you still want to argue with me, you have to admit that it is a very good one. At a very, very minimum -;)
Bibliography
[1] Java Management Extentions (JMX), https://en.wikipedia.org/wiki/Java_Management_Extensions
[2] Hans-Martin Münch — Attacking RMI based JMX services, https://mogwailabs.de/en/blog/2019/04/attacking-rmi-based-jmx-services/
[3] Marcin Wolak — CVE 2022–24082, RCE in the PEGA Platform — Discovery, Remediation & Technical Details (Long Live JMX!!!), https://marcin-wolak.medium.com/cve-2022-24082-rce-in-the-pega-platform-discovery-remediation-technical-details-long-live-69efb5437316
[4] IBM Engineering Lifecycle Management, https://www.ibm.com/products/engineering-lifecycle-management
[5] Java Remote Method Invocation (RMI), https://en.wikipedia.org/wiki/Java_remote_method_invocation
[6] Java™ Platform, Standard Edition 7 API Specification, Class LocateRegistry, https://docs.oracle.com/javase/7/docs/api/java/rmi/registry/LocateRegistry.html
[7] Nmap Network Scanner, https://nmap.org/
[8] MJET by MOGWAI LABS, https://github.com/mogwailabs/mjet
[9] Java SE 8 Documentation, https://docs.oracle.com/javase/8/docs/technotes/guides/rmi/javarmiproperties.html
