Security

Kromtech Security Center found 17 malicious docker images stored on Docker Hub for an entire year. Even after several complaints on GitHub and Twitter, research made by sysdig.com and fortinet.com, cybercriminals continued to enlarge their malware armory on Docker Hub. With more than 5 million pulls, the docker123321 registry is considered a springboard for cryptomining containers. Today’s growing number of publicly accessible misconfigured orchestration platforms like Kubernetes allows hackers to create a fully automated tool that forces these platforms to mine Monero. By pushing malicious images to a Docker Hub registry and pulling it from the victim’s system, hackers were able to mine 544.74 Monero, which is equal to $90000.

Here is the timeline:

Figure 1. Timeline of malicious docker123321 registry lifecycle.

Kubernetes clusters that were deployed for educational purposes or for tests with lack of security requirements represent a great threat for its owners. Even an experienced engineer could care less or even forget about that part of the infrastructure after tests.

Background

Palo Alto Networks post:

Attackers have traditionally profited by stealing identities or credit card numbers and then selling them on underground markets. According to Verizon Data Breach Investigations Reports, the price for stolen records has fallen, so cyber attackers are on the hunt for new ways to boost their profits. Thanks to advances in attack distribution, anonymous payments, and the ability to reliably encrypt and decrypt data, ransomware is on a tear.

With the increase in prices for cryptocurrency trendsetter and several of its altcoins, the number of cryptocurrency-mining malware incidents grew respectively. Cybercriminals have been running cryptocurrency attacks on hijacked machines for some time, finding it more profitable than ransomware. Now, however, malware authors have found a new way to take their nefarious actions into the cloud and bypass the need for hijacking individual computers. The purpose of hackers hunting for poorly configured cloud-native environments is to mine cryptocurrency using large computational power.

Why did we do this?

We noticed an increase in hacker interest in publicly accessible orchestration platforms such as Kubernetes — a container orchestration tool that automates the deployment, update, and monitoring of containers.

At the start of 2018, research by Sysdig showed that attackers moved on from EC2 exploits to container-specific and kubernetes-specific exploits. A preconfigured Kubernetes instance located on honeypot servers was poisoned with malicious Docker containers that would mine Monero.

Cryptojaking has become a real-life issue, targeting a diverse array of victims, from individual consumers to large manufacturers. In February 2018, Checkpoint researchers found one of the biggest malicious mining operations ever discovered. Cybercriminals exploited the known CVE-2017-1000353 vulnerability in the Jenkins Java deserialization implementation. Since Jenkins has been called the most widely deployed automation server with an estimated 1 million users, the attack resulted in way more serious consequences. During malicious mining operation, the hackers have accumulated 10,800 Monero, which is currently worth $3,436,776.

Around the same time, in February 2018, RedLock researchers discovered hundreds of Kubernetes administration consoles accessible over the internet without any password protection, including servers belonging to Tesla. The hackers had infiltrated Tesla’s Kubernetes console, which was not password protected. Within one Kubernetes pod, access credentials were exposed to Tesla’s AWS environment, which contained an Amazon S3 (Amazon Simple Storage Service) bucket that had sensitive data such as telemetry. In addition to the data exposure, hackers were performing crypto mining from within one of Tesla’s Kubernetes pods.

The Tesla incident is just the first of many container technology-based exploits we will see in the coming months and years.

What are containers, Docker and Kubernetes?

Containers are a way of packaging software. You can think of running a container like running a virtual machine, without the overhead of spinning up an entire operating system.

Docker helps you create and deploy software within containers. With Docker, you create a special file called a Dockerfile. Dockerfiles define a build process, which, when fed to the ‘docker build’ command, will produce an immutable docker image. You can think of this as a snapshot of your application, ready to be brought to life at any time. When you want to start it up, just use the ‘docker run’ command to run it anywhere the docker daemon is supported and running. It can be on your laptop, your production server in the cloud, or on a raspberry pi. Docker also provides a cloud-based repository called Docker Hub. You can think of it like GitHub for Docker Images. You can use Docker Hub to store and distribute the container images you build.

When you need to start the right containers at the right time, figure out how they can talk to each other, handle storage considerations, and deal with failed containers or hardware, that’s where Kubernetes comes in. Kubernetes is an open source container orchestration platform, allowing large numbers of containers to work together in harmony, reducing operational burden. It helps with things like:

Running containers across many different machines

Scaling up or down by adding or removing containers when demand changes

Keeping storage consistent with multiple instances of an application

Distributing load between the containers

Launching new containers on different machines if something fails

Kubernetes is supported by all major container management and cloud platforms such as Red Hat OpenShift, Docker EE, Rancher, IBM Cloud, AWS EKS, Azure, SUSE CaaS, and Google Cloud.

How cybercriminals behave

Both original attack schemes on Docker engine and Kubernetes instances were explained by Aqua Security and Alexander Urcioli respectively.

In the first case, researchers from Aqua Security simulated a system with an “accidentally” exposed docker daemon. Here is what they discovered two days later:

Hundreds of suspicious actions were logged, many of them were created automatically.

The attacker attempted to execute a variety of docker commands for image and container management.

After successful information gathering about the running Docker version, the attacker used the docker import functionality for image injection.

After a successful image injection, the attacker would start mining.

The second case shows how Alexander Urcioli came across an already compromised personal Kubernetes cluster. He realized that due to misconfiguration that resulted in the public exposing of the kubelet ports (TCP 10250, TCP 10255) and unauthenticated API requests, the attacker:

Sent two requests: an initial POST and a follow-up GET with exec command to kubelet.

Executed a dropper script on a running Docker container through kubelet. Dropper script named “kube.lock” would download the mining software from transfer.sh and execute it.

Recently we found another disturbing issue with misconfigured Kubernetes cluster. It turns out that the kubelet exposes an unauthenticated endpoint on port 10250.

Let’s come back Alexander Urcioli research one more time:

There are two ports that kubelet listens in on, 10255 and 10250. The former is a read-only HTTP port and the latter is an HTTPS port that can essentially do whatever you want.

Further inspection showed that Kubernetes PodList leaked AWS Access keys (access key ID and secret access key), which simply provide a root access to AWS environments including an Amazon EC2, RDS, S3, and related actions on them.

When we look through latest kubelet documentation we find debug handlers in charge of running code in any container. The option is enabled by default.

The option left on by default, in conjunction with exposed 10250 port, could have led to devastating consequences.

We can assume now which steps an average cybercriminal can take to attack container based virtualized environments:

Collect targets automatically through Shodan, Censys or Zoomeye.

Infiltrate vulnerable or misconfigured Docker registries or Kubernetes instances.

Exploit weak default settings and inject mining malware within containers. Usually, this is done by injecting a malicious docker image into the docker host. The popular and conventional way to do this is to push the image to a registry (Docker Hub is the natural place) and pull it from the victim host.

It all also requires C2 servers, how cybercriminals build it:

Collect targets automatically through Shodan, Censys or Zoomeye.

Automate the exploitation of remote targets using something like AutoSploit.

Take full control of compromised targets and place C2 servers there.

Does Docker care?

Why it is feasible to pack mining malware into Docker containers? We decided to poke around Docker images with an eye to security aspects.

In an interview, Ericsson’s Head of Cloud Jason Hoffman stated: “Docker’s taking off because it’s the new package management”. That provides a good explanation of Docker’s rapid adoption, but also hides the fact that Docker images are generally dependent on the package manager provided by an underlying Linux distribution. Images like CentOS 5.11 are deliberately held back for the sake of compatibility and have the Shellshock vulnerability.

From https://medium.com/microscaling-systems/dockerfile-security-tuneup-166f1cdafea1

One of the key differences between containers and virtual machines is that containers share the kernel with the host. By default, docker containers run as root which causes a breakout risk. If your container becomes compromised as root it has root access to the host.

Docker is making Security Scanning available as a free preview for a limited time. From the Docker docs:

Docker Security Scanning

The Docker Security Scanning preview service will end on March 31st, 2018, for private repos (not official repos) in both Docker Cloud and Docker Hub. Until then, scanning in private repos is limited to one scan per day on the “latest” tag.

Relying on blog.docker.com:

Docker Security Scanning went alongside Docker Cloud to trigger a series of events once a new image is pushed to a repository. The service included a scan trigger, the scanner, a database, plugin framework and validation services that connect to CVE databases.

Security Scanning provides a detailed security profile of your Docker images for proactive risk management and to streamline software compliance. Docker Security Scanning conducts binary level scanning of your images before they are deployed, provides a detailed bill of materials (BOM) that lists out all the layers and components, continuously monitors for new vulnerabilities, and provides notifications when new vulnerabilities are found.

From the Docker docs:

Cluster and application management services in Docker Cloud are shutting down on May 21. You must migrate your applications from Docker Cloud to another platform and deregister your Swarms. The Docker Cloud runtime is being discontinued. This means that you will no longer be able to manage your nodes, swarm clusters, and the applications that run on them in Docker Cloud. To protect your applications, you must migrate them to another platform, and if applicable, deregister your Swarms from Docker Cloud.

It seems that the Docker ecosystem is becoming more enterprise oriented and the responsibility for safe migration and further secure maintenance falls on ordinary developers.

Conclusions

For ordinary users, just pulling a Docker image from the DockerHub is like pulling arbitrary binary data from somewhere, executing it, and hoping for the best without really knowing what’s in it.

The main thing we should consider is traceability. The process of pulling a Docker image has to be transparent and easy to follow. First, you can simply try to look through Dockerfile to find out what the FROM and ENTRYPOINT notations are and what the container does. Second, Docker images are built using the Docker automated builds. That’s because, with Docker automated builds, you get traceability between the source of the Dockerfile, the version of the image, and the actual build output.

Each build´s details show a lot of information that can be used for improved trust in the image:

The SHA from the git repository with Dockerfile

Every command from the Dockerfile that is executed is shown

Finally, it all ends with a digest of the image pushed

Kubernetes deployments are just as vulnerable to attacks and exploits from hackers and insiders as traditional environments. By attacking the orchestration tools, hackers can disrupt running applications and even gain control of the underlying resources used to run containers. Old models and tools for security will not be able to keep up in a constantly changing container environment. You need to ask yourself whether you’re able to monitor what’s going on inside a pod or container to determine if there is a potential exploit. Pay specific attention to the most damaging “kill chain” attacks — a series of malicious activities which together achieve the attacker’s goal. Detecting events in a kill chain requires multiple layers of security monitoring. The most critical vectors to monitor in order to have the best chances of detection in a production environment are Network inspection, Container monitoring, and Host security.

An internal and external communication within Kubernetes cluster should be considered as most important part of the secure configuration. The key notions we learned:

The connection is not secure enough to be run across the internet.kubelet

SSH tunnels must be used to securely put packets onto the cluster's network without exposing the kubelet's web server to the internet.

The kubelet needs to serve its https endpoint with a certificate that is signed by the cluster CA.

Adherence to these principles can help you gain a certain level of security awareness.

ABOUT MACKEEPER

With MacKeeper, we aim to make using your Mac easier and safer through reliable technology solutions. MacKeeper comes with the essentials to clean up and speed up your Mac and make your online experience more private and secure.