The ultimate guide to scanning Kubernetes workloads using Trivy-Operator
Testing is a vital aspect of security. Unfortunately, there are vulnerabilities that bypass the testing stage and introduce flaws in the production environment.
Kubernetes vulnerabilities make more impact when exploited on running applications and production environments. These vulnerabilities can allow hackers to exploit the host machine if the container has escalated privileges. Therefore, it is important to carry out primary checks that ensure that Kubernetes doesn’t have any security vulnerabilities.
Fortunately, there is a company called Aqua Security, a cloud-native security provider that creates multiple free and premium Kubernetes security tools. One of the Kubernetes security tools developed by AquaSecurity is Trivy-Operator, which is a kubectl plugin used to scan workloads for vulnerabilities. After scanning the workloads, Trivy-Operator will give the security performance report of the workload.
Trivy-Operator will give you workload security observability which is crucial when detecting and eliminating bugs. In this tutorial, you will learn how to install Trivy-Operator and scan Kubernetes workloads using Trivy-Operator.
You need kubectl and a running Kubernetes cluster.
How to install Trivy-Operator
In May 2022, Trivy Operator replaced the obsolete Starboard workload scanner. Trivy-Operator is available on all three operating systems. Use the following command to install Trivy-Operator using kubectl:
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/trivy-operator/v0.1.9/deploy/static/trivy-operator.yaml
You will get the output that ends with the following:
role.rbac.authorization.k8s.io/trivy-operator created rolebinding.rbac.authorization.k8s.io/trivy-operator created service/trivy-operator created
Use the following command to check if the Trivy-Operator plugin has been installed successfully:
kubectl get deployment -n trivy-system
You will get the following output:
NAME READY UP-TO-DATE AVAILABLE AGE trivy-operator 1/1 1 1 2m3s
If you want to upgrade Trivy-Operator plugin later on, start by uninstalling the current version using this command:
kubectl delete -f https://raw.githubusercontent.com/aquasecurity/trivy-operator/v0.1.9/deploy/static/trivy-operator.yaml
After uninstalling it, use the first command you used to install the Trivy-Operator to update it. Currently, there isn’t an automatic way of updating the Trivy-Operator.
Creating a deployment to scan using Trivy-Operator
Kubernetes workloads are running applications. Trivy-Operator only scans workloads, such as deployments that deploy applications to the production and staging environment. So, to learn how to use Trivy-Operator, we will create a deployment resource. To do that, create a YAML file called
mattermost-deployment.yaml and add the following contents:
apiVersion: apps/v1 kind: Deployment metadata: name: mattermost-app labels: app: boemo-app spec: replicas: 1 selector: matchLabels: app: boemo-app template: metadata: labels: app: boemo-app spec: containers: - name: server image: nginx:1.17 volumeMounts: - name: boemo-app mountPath: /usr/share/nginx/html ports: - containerPort: 80 protocol: TCP resources: requests: cpu: 100m memory: "128M" limits: cpu: 100m memory: "256M" env: - name: LOG_LEVEL value: "DEBUG" volumes: - name: boemo-app configMap: name: boemo-app items: - key: body path: index.html
After creating the deployment.yaml file, apply it to your cluster:
kubectl apply -f mattermost-deployment.yaml
You will get the following output:
Fetching Trivy-Operator scan results
After creating the previous deployment, Trivy-Operator will automatically detect and scan it before creating a configuration audit report.
Configuration audits assess the health of the workload configuration. Use the following command to fetch all configuration reports:
kubectl get configauditreports -o wide
You will get the following output:
NAME SCANNER AGE CRITICAL HIGH MEDIUM LOW replicaset-mattermost-app-59d45 Trivy 4m33s 0 0 2 6
Now, we know that the deployment we created earlier on has a configuration report. Use the following command to retrieve the configuration scan results for your replicset (in my case, that’s
kubectl describe configauditreport replicaset-mattermost-app-59d4567b6b
Let’s look at the resulting output, starting with this:
Manager: trivy-operator Operation: Update Time: 2022-09-05T22:30:44Z Owner References: API Version: apps/v1 Block Owner Deletion: false Controller: true Kind: ReplicaSet Name: mattermost-app-59d4567b6b UID: 3f263b27-6527-49b1-ae7b-1e332bd562bf Resource Version: 73504 UID: c703c5b7-8ee6-4c30-aef3-9342f6abe047
The above section of the scan results explains the date and time when the security audit was carried out. It also specifies the API version and resource kind being scanned.
The next section of the scan results shows you a list of detected vulnerabilities.
Below is an example of the detected vulnerability; its severity level is high because the detected vulnerability is “Access to host PID.” The vulnerability is very serious because the attacker will be able to access the host machine in case of a security breach, thus escalating the impact. Vulnerabilities classified as CRITICAL and HIGH should be resolved as soon as possible.
Severity: HIGH Success: true Title: Access to host PID Category: Kubernetes Security Check Check ID: KSV105 Description: Containers should be forbidden from running with a root UID. Messages:
The description field tells you what shouldn’t be done to prevent the vulnerability from occurring. Here’s another example of a vulnerability that has a LOW severity:
Severity: LOW Success: true Title: Containers must not set runAsUser to 0 Category: Kubernetes Security Check Check ID: KSV015 Description: When containers have resource requests specified, the scheduler can make better decisions about which nodes to place pods on, and how to deal with resource contention. Messages:
Learn more about Kubernetes security
Hackers use simple techniques to jeopardize the Kubernetes environment when companies and system administrators aren’t putting in the time and resources required to secure the Kubernetes runtime. Simple techniques, such as stealing secrets from an unencrypted etcd, become very common. Remember, hackers are working 24/7 to find Kubernetes weaknesses and vulnerabilities.
That being the case, Kubernetes workload security observability has to be mandatory and added to your DevOps workflow. Knowing what vulnerabilities are constantly jeopardizing your workload will help you to eliminate the chances of a security breach happening.
To learn more about what you can do to keep Kubernetes secure, check out this post: The Top 7 Open Source Tools for Securing Your Kubernetes Cluster.
This blog post was created as part of the Mattermost Community Writing Program and is published under the CC BY-NC-SA 4.0 license. To learn more about the Mattermost Community Writing Program, check this out.