The ultimate guide to scanning Kubernetes workloads using Trivy-Operator

Testing is a vital aspect of security. Unfortunately, there are vulnerabilities that bypass the testing stage and introduce flaws in the production environment. 

Kubernetes vulnerabilities make more impact when exploited on running applications and production environments. These vulnerabilities can allow hackers to exploit the host machine if the container has escalated privileges. Therefore, it is important to carry out primary checks that ensure that Kubernetes doesn’t have any security vulnerabilities.

Fortunately, there is a company called Aqua Security, a cloud-native security provider that creates multiple free and premium Kubernetes security tools. One of the Kubernetes security tools developed by AquaSecurity is Trivy-Operator, which is a kubectl plugin used to scan workloads for vulnerabilities. After scanning the workloads, Trivy-Operator will give the security performance report of the workload.

Trivy-Operator will give you workload security observability which is crucial when detecting and eliminating bugs. In this tutorial, you will learn how to install Trivy-Operator and scan Kubernetes workloads using Trivy-Operator.

Prerequisites

You need kubectl and a running Kubernetes cluster.

How to install Trivy-Operator

In May 2022, Trivy Operator replaced the obsolete Starboard workload scanner. Trivy-Operator is available on all three operating systems. Use the following command to install Trivy-Operator using kubectl:

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/trivy-operator/v0.1.9/deploy/static/trivy-operator.yaml

You will get the output that ends with the following:

role.rbac.authorization.k8s.io/trivy-operator created
rolebinding.rbac.authorization.k8s.io/trivy-operator created
service/trivy-operator created

Use the following command to check if the Trivy-Operator plugin has been installed successfully:

kubectl get deployment -n trivy-system

You will get the following output:

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
trivy-operator   1/1     1            1           2m3s

If you want to upgrade Trivy-Operator plugin later on, start by uninstalling the current version using this command:

kubectl delete -f https://raw.githubusercontent.com/aquasecurity/trivy-operator/v0.1.9/deploy/static/trivy-operator.yaml

After uninstalling it, use the first command you used to install the Trivy-Operator to update it. Currently, there isn’t an automatic way of updating the Trivy-Operator.

Creating a deployment to scan using Trivy-Operator

Kubernetes workloads are running applications. Trivy-Operator only scans workloads, such as deployments that deploy applications to the production and staging environment. So, to learn how to use Trivy-Operator, we will create a deployment resource. To do that, create a YAML file called mattermost-deployment.yaml and add the following contents:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mattermost-app
  labels:
    app: boemo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: boemo-app
  template:
    metadata:
      labels:
        app: boemo-app
    spec:
      containers:
      - name: server
        image: nginx:1.17
        volumeMounts:
          - name: boemo-app
            mountPath: /usr/share/nginx/html
        ports:
        - containerPort: 80
          protocol: TCP
        resources:
          requests:
            cpu: 100m
            memory: "128M"
          limits:
            cpu: 100m
            memory: "256M"
        env:
        - name: LOG_LEVEL
          value: "DEBUG"
      volumes:
      - name: boemo-app
        configMap:
          name: boemo-app
          items:
          - key: body
            path: index.html

After creating the deployment.yaml file, apply it to your cluster:

kubectl apply -f mattermost-deployment.yaml

You will get the following output:

deployment.apps/mattermost-app created

Fetching Trivy-Operator scan results

After creating the previous deployment, Trivy-Operator will automatically detect and scan it before creating a configuration audit report. 

Configuration audits assess the health of the workload configuration. Use the following command to fetch all configuration reports:

kubectl get configauditreports -o wide

You will get the following output:

NAME                           SCANNER AGE  CRITICAL HIGH MEDIUM  LOW
replicaset-mattermost-app-59d45 Trivy  4m33s  0       0    2        6

Now, we know that the deployment we created earlier on has a configuration report. Use the following command to retrieve the configuration scan results for your replicset (in my case, that’s replicaset-mattermost-app-59d4567b6b):

kubectl describe configauditreport replicaset-mattermost-app-59d4567b6b

Let’s look at the resulting output, starting with this:

  Manager:    trivy-operator
    Operation:  Update
    Time:       2022-09-05T22:30:44Z
  Owner References:
    API Version:           apps/v1
    Block Owner Deletion:  false
    Controller:            true
    Kind:                  ReplicaSet
    Name:                  mattermost-app-59d4567b6b
    UID:                   3f263b27-6527-49b1-ae7b-1e332bd562bf
  Resource Version:        73504
  UID:                     c703c5b7-8ee6-4c30-aef3-9342f6abe047

The above section of the scan results explains the date and time when the security audit was carried out. It also specifies the API version and resource kind being scanned. 

The next section of the scan results shows you a list of detected vulnerabilities. 

Below is an example of the detected vulnerability; its severity level is high because the detected vulnerability is “Access to host PID.” The vulnerability is very serious because the attacker will be able to access the host machine in case of a security breach, thus escalating the impact. Vulnerabilities classified as CRITICAL and HIGH should be resolved as soon as possible.

 Severity:     HIGH
    Success:      true
    Title:        Access to host PID
    Category:     Kubernetes Security Check
    Check ID:     KSV105
    Description:  Containers should be forbidden from running with a root UID.
    Messages:

The description field tells you what shouldn’t be done to prevent the vulnerability from occurring. Here’s another example of a vulnerability that has a LOW severity:

 Severity:     LOW
    Success:      true
    Title:        Containers must not set runAsUser to 0
    Category:     Kubernetes Security Check
    Check ID:     KSV015
    Description:  When containers have resource requests specified, the scheduler can make better decisions about which nodes to place pods on, and how to deal with resource contention.
    Messages:

Learn more about Kubernetes security

Hackers use simple techniques to jeopardize the Kubernetes environment when companies and system administrators aren’t putting in the time and resources required to secure the Kubernetes runtime. Simple techniques, such as stealing secrets from an unencrypted etcd, become very common. Remember, hackers are working 24/7 to find Kubernetes weaknesses and vulnerabilities. 

That being the case, Kubernetes workload security observability has to be mandatory and added to your DevOps workflow. Knowing what vulnerabilities are constantly jeopardizing your workload will help you to eliminate the chances of a security breach happening.

To learn more about what you can do to keep Kubernetes secure, check out this post: The Top 7 Open Source Tools for Securing Your Kubernetes Cluster.

This blog post was created as part of the Mattermost Community Writing Program and is published under the CC BY-NC-SA 4.0 license. To learn more about the Mattermost Community Writing Program, check this out.