The Top 7 Open Source Tools for Securing Your Kubernetes Cluster

February 1, 2022

Engineering

This article explores how to secure production Kubernetes clusters with the help of open source tools. As a prerequisite, you’ll need to have basic beginner-level knowledge of Docker and Kubernetes.

In a nutshell, Kubernetes is a container orchestration tool and Docker is a containerization platform. Some of the most famous Kubernetes clusters managed by cloud providers include AWS EKS, Azure AKS, and Google CKE.

In the following sections, we’ll discuss various open source tools that are useful for securing Kubernetes clusters, including code snippets that will help with static scanning of Docker images, security auditing, hardening Kubernetes clusters, and implementing runtime security.

Static Scanning of Docker Images

Dockerfile is a plain text file that contains a set of instructions that provides the specifications for creating a Docker image. Additionally, containers are the running instances of Docker images and Kubernetes supports Docker runtime.

It’s highly recommended to scan your Docker images for security vulnerabilities before pushing them to a Kubernetes runtime environment. This will help you avoid supply chain attacks and improve security by shifting left.

docker scan

Docker Engine has now integrated Synk’s scanning capabilities to identify vulnerabilities in the images. The output of the scan is the list of CVEs and recommendations.

Below are the open source tools that can be used to perform security scans that can be integrated into your CI/CD pipeline to scan the images while building your applications:

Trivy, a scanning tool that provides vulnerability detection for operating systems
Clair, a vulnerability static scanning tool for containers
Kube-bench, a tool that determines whether Kubernetes is deployed optimally
kubeaudit, a tool that audits Kubernetes deployments against common security controls
Kubescape, a tool that checks whether Kubernetes is deployed in accordance with major compliance frameworks
kube-hunter, a penetration testing tool that discovers and exploits vulnerabilities
Sysdig Falco, a runtime security solution for risk and threat detection across Kubernetes clusters

1. Vulnerability Detection with Trivy

Trivy is an open source scanning tool that’s an excellent choice for teams looking to implement static application security testing and quick scans. It provides comprehensive vulnerability detection for operating systems, infrastructure as code files, and language packages and is also used to detect configuration issues. Trivy also auto-updates its vulnerability database

After conducting a Trivy scan, you’ll end up with a list of vulnerabilities, their severity, and a CVE number, if one exists.

Installation

This script will download and install Trivy based on your operating system.

curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin v0.21.1

Scanning and output

trivy image --severity CRITICAL,HIGH apache/airflow:3.4-alpine

The above command scans Apache Airflow images for CRITICAL & HIGH severity vulnerabilities. Once complete, the scan report looks like this:

To learn more about using an OPA policy with Trivy to handle container vulnerabilities, check this out.

2. Vulnerability Static Analysis with Clair

Clair is an open source vulnerability static scanning tool for containers. The tool has multiple deployment modes and is best suited for high scalability and availability. Clair supports REST APIs and provides HTML scan reports. The Amazon Elastic Container Registry (Amazon ECR) uses the CVEs database from the Clair project and provides a list of findings.

Installation and scanning

In this example, we’re using the arminc/clair-db Docker image that’s preloaded with CVE data. clair-scanner will connect with the Clair endpoint over the local network.

docker run -d --name db arminc/clair-db:latest  
docker run -d --link db:postgres --name clair arminc/clair-local-scan:v2.0.6  
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock --network=container:clair ovotech/clair-scanner clair-scanner --threshold='Critical' apache/airflow:2.0.2

Once a Clair scan is complete, you’ll see something like this:

You can also get the output reports in JSON format if you prefer.

In most situations, the best solution for resolving vulnerabilities reported by Clair is to either update the underlying operating system with the built-in package manager or to update to the latest version of your Docker images. If the volume of vulnerable packages is too much to manage, consider using a distroless base image.

Securing & Auditing Your Kubernetes Deployment

In order to improve the security posture of Kubernetes, you need to regularly audit your deployment and perform security scans on the setup as per security benchmarks. This section is all about auditing and securing your Kubernetes clusters using various open source tools.

3. CIS Benchmarking with Kube-bench

Kube-bench is an open source tool that checks if Kubernetes is deployed optimally according to the CIS Kubernetes Benchmark, which contains a set of Kubernetes security best practices. As such, kube-bench is best when required to scan only for CIS benchmarking purposes.

You can run kube-bench inside a pod. The GitHub repository contains cloud-specific job-<cloud_provider>.yaml files, and kube-bench will automatically determine which test set to run based on the Kubernetes version running on the machine.

Running kube-bench

git clone https://github.com/aquasecurity/kube-bench.git  
cd kube-bench  
kubectl apply -f job.yaml

Kubernetes benchmark for security — Source: https://github.com/aquasecurity/kube-bench/

In order to mitigate any issues kube-bench reports, refer to the official Kubernetes CIS benchmarking documentation, which contains details and remediation instructions for every finding.

4. Auditing Kubernetes with kubeaudit

Made by Shopify, kubeaudit is an open source tool that audits Kubernetes deployments against common security controls like:

run as non-root
use a read-only root filesystem
drop scary capabilities, don’t add new ones
don’t run privileged

Installation and execution

go get -v github.com/Shopify/kubeaudit

Kubeaudit can run in three different modes: manifest, cluster, and local. It can also autofix the manifest, which sets it apart from other competitors.

Running in manifest mode – In this mode, you need to provide the manifest file of the relevant Kubernetes resource.

kubeaudit all -f "/path/to/manifest.yml"  
kubeaudit autofix -f "/path/to/manifest.yml"

Once that’s done, you’ll see something like this:

Running in cluster mode – If kubeaudit is running within a container in a cluster, it will automatically audit all resources in that cluster.

kubeaudit all

Running in local mode – When kubeaudit is running on a local machine, it will try to connect to the Kubernetes cluster by fetching details from the $HOME/.kube/config file by default. Additionally, you can also provide a path for Kube config:

kubeaudit all -f "/path/to/config"

Additionally, kubeaudit has multiple auditing profiles, including apparmor, capabilities, limits, privileged, rootfs, seccomp, netpols, and asat.

5. Security Testing with Kubescape

Kubescape is an open source tool that checks if Kubernetes is deployed in accordance with almost all major compliance frameworks, including the NSA-CISA and MITRE ATT&CK®, as well as DevSecOps best practices. Kubescape can also be integrated with CI tools.

Installation

curl -s https://raw.githubusercontent.com/armosec/kubescape/master/install.sh | /bin/bash

Running

kubescape scan framework mitre --submit --format json --output results.json  
kubescape scan framework nsa --submit --include-namespaces development,staging,production

After running Kubescape, you should see something like this:

Source: https://github.com/armosec/kubescape

6. Penetration Testing with kube-hunter

Written in Python, kube-hunter is an open source penetration testing tool that enables you to write custom modules that can be executed from local machines, inside the cluster, and remotely in both active and passive mode.

In active mode, kube-hunter will discover and further exploit any vulnerabilities.

pip install kube-hunter # Installing Kube-hunter.  
kube-hunter --remote some.node.com # Running remotely.  
kube-hunter --cidr 192.168.0.0/24. # Network Scanning.  
kube-hunter --remote some.domain.com --active # Active Mode.  
kube-hunter --list --active # List of test cases.  
kube-hunter --remote some.node.com --json # Json output  
kube-hunter --k8s-auto-discover-nodes --kubeconfig "/path/config"

Further, you can also run kube-hunter on a staging environment as a malicious pod that can act as a real attack simulation.

The previous three tools — kubeaudit, Kubescape, and kube-hunter — each provide a detailed report about misconfigurations, vulnerabilities, missing checks, insecure practices, and more. It’s highly recommended to follow some Kubernetes security hardening guidelines to improve your security posture over time and validate it with these tools. For more, check out the NSA and CISA’s recently released Kubernetes hardening guidance.

7. Runtime Security with Sysdig Falco

Sysdig Falco is an open source runtime security solution used for continuous risk and threat detection across Kubernetes clusters. The tool acts as a security camera that continuously detects unexpected behavior, configuration changes, intrusions, and data theft in real time. Currently, Sysdig Falco is the only CNCF-approved open source solution for Kubernetes runtime security.

Falco — which enables you to write custom plugins and has all the major features found in its commercial competitors (e.g., Aquasec and Twistlock) — consumes the raw stream of system call information using drives like a kernel module or an eBPF probe. Falco can run as a daemonset on Kubernetes Nodes, where it can monitor and transmit real-time security alerts in JSON format via stdout, HTTP hooks, and syslog.

Installation with HELM

You can install the Sysdig Falco daemonset with custom parameters using a Helm chart:

helm repo add falcosecurity https://falcosecurity.github.io/charts  
helm repo update  
helm install falco -f values.yaml falcosecurity/falco

Falco Security Rules

Falco provides more than 80 default security rules for runtime security, Kubernetes control plane audits, and applications security. You can customize the provided YAML-based rules in accordance with your own requirements. Here’s a table that explains some of the more common rules:

Rule name	Tags
Detecting CryptoJacking	Mitre
Read Sensitive File Untrusted	File Integrity Monitoring
Launch of Privileged Containers	CIS
Suspicious Network Tool in Container	Network, Mitre
Launch Remote File Copy Tools in Container	Exfiltration

Falco Security Alerts on Mattermost

Falcosidekick is a daemon that collects Falco Logs and transmits them to various channels — e.g., Mattermost, Elasticsearch, S3, and Kafka.

helm install falco falcosecurity/falco -f values.yaml --set falcosidekick.enabled=true --set falcosidekick.webui.enabled=true

We can configure Mattermost in falcosidekick chart’s values.yaml file; you’ll need to leverage the incoming webhooks feature of Mattermost to take advantage of this.

config:  
    mattermost:  
    webhookurl: "" # (ex: http://XXXX/hooks/YYYY)  
    #footer: "" # Mattermost footer  
    #icon: "" # Mattermost icon (avatar)  
    #username: "" # Mattermost username (default: Falcosidekick)  
    outputformat: "all"  
    minimumpriority: "debug"  
    # messageformat: "**{{ .Rule }}** triggered by user **{{ index .OutputFields \"user.name\" }}**".  
    # mutualtls: false  
    # checkcert: true

Once enabled, security alerts on Mattermost Channels will look like this:

Falcosidekick alerts in Mattermost — Source

Falco is required to automatically respond to the security alerts it receives — like removing a malicious file, deleting an infected pod, and applying a firewall rule. The tool can be integrated with active response with Kubeless, Knative, and Argo, among others.

For more information, read this.

Conclusion

As you can see, there’s a very rich ecosystem of open source tools you can use to improve Kubernetes security. There are multiple tools available on GitHub that enable you to perform automated scans, CI integrations, runtime security, penetration testing, and audit checks. Use this article as a reference guide as you implement your Kubernetes security program and keep your applications, network, and data secure.

This blog post was created as part of the Mattermost Community Writing Program and is published under the CC BY-NC-SA 4.0 license. To learn more about the Mattermost Community Writing Program, check this out.