Uncovering a reverse shell attack in real time

Date: 2025-04-12

I purchased the LFS260: Kubernetes Security Essentials course about a month ago and was working through the course material recently in preparation for the Certified Kubernetes Security Specialist (CKS) exam. The course included plenty of hands-on lab exercises covering each topic in detail which I followed through by creating a 2-node kubeadm cluster on AWS with Amazon EC2.

The environment was configured with what I thought were reasonably secure defaults for my own purposes of experimenting with and learning about Kubernetes security, but since it was just for personal use, I made some trade-offs between security and convenience:

1. Both nodes were placed in a public subnet with unrestricted outbound Internet connectivity
2. Both nodes allowed unrestricted inbound SSH access
3. Both nodes had SSH public key authentication configured with no login password set
4. Both nodes were initially deployed 2 weeks ago from the latest official Ubuntu 24.04 LTS AMI
5. No security hardening or package updates were performed on either node at the OS level
6. Limited security hardening was performed on the cluster level based on the CIS Kubernetes Benchmarks while following through the course exercises

One of the course exercises involved installing an eBPF-based security tool by Aqua Security known as Tracee and observing the JSON events emitted which uncover the runtime activity of applications running on both nodes in the cluster. Initially, the JSON events appeared normal and the activity matched the expected output based on the actions performed in the exercise. At the end of the exercise, I kept Tracee installed and running on my cluster to compare its output to those generated by Falco in the next exercise.

While comparing the output of both runtime security tools, an interesting event emitted by Tracee caught my eye. Soon did I realize that the worker node in my cluster was compromised!

A reverse shell attack!

Here’s the JSON event of the reverse shell attack in all its glory.

A few important pieces of information apparent from the JSON event:

1. An outbound connection was made to the public IP address 92.118.39.76 from/to a random high-numbered port
2. Based on the event description, the process had its I/O redirected to a socket indicative of a reverse shell attack
3. The timestamp of the event was only about 15 minutes before I noticed the event (not shown in the image)
4. The affected host was my worker node which I only logged in to 2 weeks ago during the initial cluster setup (not shown in the image)

The last observation in particular caught my attention so I decided to check the IP address against an IP reputation checker. After some initial research, I settled with the Symantec online IP reputation checker by Broadcom. I entered the IP address and clicked “submit”, hoping it would inform me that it was just a normal IP address with no bad reputation associated with it. But voila - it responded unambiguously with the message:

The IP Address 92.118.39.76 was found to have a negative reputation. Reasons for this assessment include:

• The host has been observed sending spam in a format that is similar to snow shoe spamming techniques.
• The host is unauthorized to send email directly to email servers.

By now, it was apparent that my worker node was pwned!

Lessons learned

While I knew that allowing unrestricted SSH access from the Internet was far from a secure configuration and would never do that in production, I had assumed that using a recent official base AMI with the latest patches and enabling SSH public key authentication instead of password authentication would suffice to keep most attackers at bay. Reality proved that my mindset could not be further from the truth - the attack succeeded within 1 hour of starting my cluster nodes! Fortunately, I observed the attack happening in real time with a runtime security and observability tool such as Tracee, as otherwise I wouldn’t even be aware that such an attack had already taken place and succeeded.

This incident demonstrates the importance of adopting a security-first mindset and implementing proper security measures at every level, from your infrastructure all the way up to your applications - “defense in depth” as some would put it.

Subscribe:

Return to homepage