At Mattermost, we aim to develop a product that is simple to deploy and manage.

To this end, we recently examined Kubernetes, today’s most popular orchestration platform that uses containers. 

After determining the platform would make it even easier to deploy, manage and scale our product, we decided to migrate our “virtual office”—our Mattermost instance where we interact with customers and members of our community—from AWS to Kubernetes. That way, we’d be able to see what improvements we needed to make in order to make our platform more cloud-native.

Here’s how we did the migration:

Let’s explore each step in detail.

1. Decide where to run Kubernetes

First, we installed Kubernetes on the latest version of Mattermost.

Next, we needed to figure out where to run Kubernetes. We had several options, including on-premises, AWS, Azure, GCP or DigitalOcean.

We decided to go with AWS EKS because we currently don’t have the overhead to manage the entire Kubernetes cluster, only the worker nodes; AWS manages the master nodes.

(For the purposes of this post, we won’t focus on how to deploy Kubernetes on other platforms. If you need help with that, check this out.)

2. Set up and configure Ingress

After we deployed our Kubernetes cluster, we installed NGINX Ingress using this documentation.

To reduce WebSocket errors, we added cache and keep-alive in our ConfigMap:

`kind: ConfigMap apiVersion: v1 metadata:  name: nginx-configuration  namespace: ingress-nginx  labels:    app.kubernetes.io/name: ingress-nginx    app.kubernetes.io/part-of: ingress-nginx data:  use-proxy-protocol: "true"  http-snippet: "proxy_cache_path /cache/mattermost levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;"  keep-alive: "3600"`

The Kubernetes service will look like this:

`kind: Service apiVersion: v1 metadata:  name: ingress-nginx  namespace: ingress-nginx  labels:    app.kubernetes.io/name: ingress-nginx    app.kubernetes.io/part-of: ingress-nginx  annotations:    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600" spec:  type: LoadBalancer  selector:    app.kubernetes.io/name: ingress-nginx    app.kubernetes.io/part-of: ingress-nginx  ports:    - name: http      port: 80      targetPort: http    - name: https      port: 443      targetPort: https`

We also needed to add volume to store the cache to the deployment. So we updated the deployment manifest:

`...      volumeMounts:      - mountPath: /cache/mattermost        name: mattermost-cache ...      volumes:      - emptyDir: {}        name: mattermost-cache`

To get the TLS certificates—which are not required but are recommended for security reasons—we installed a cert manager using the helm chart. For more information, follow the instructions here.

3. Set up the database

In our legacy deployment (i.e., running servers on EC2), we had our database set up in AWS using RDS, a managed service by AWS.

While we transitioned from EC2 to Kubernetes, we decided to keep using RDS for our database. We do, however, plan to move the database to Kubernetes as well. It’s only a matter of time.

Here, we had to make a few configuration changes to the security groups and VPCs in order to allow the Kubernetes cluster to talk to the database.

4. Set up Mattermost

To set up Mattermost Enterprise Edition, we first had to set up a deployment manifest. (If you use Team Edition, use the official Mattermost helm chart for help with installation.)

Here’s what our manifest looks like:

` apiVersion: apps/v1 kind: Deployment metadata:  name: mattermost-community-daily-app  labels:    app.kubernetes.io/name: mattermost-community-daily-app    app.kubernetes.io/component: "mattermost-community-daily-app" spec:  replicas: 2  strategy:    type: RollingUpdate    rollingUpdate:      maxUnavailable: 1  selector:    matchLabels:      app.kubernetes.io/name: mattermost-community-daily-app      app.kubernetes.io/component: "mattermost-community-daily-app"  template:    metadata:      labels:        app.kubernetes.io/name: mattermost-community-daily-app        app.kubernetes.io/component: "mattermost-community-daily-app"      annotations:        prometheus.io/scrape: "true"        prometheus.io/port: "8067"        prometheus.io/path: "/metrics"    spec:      initContainers:      - name: init-config        image: busybox        imagePullPolicy: IfNotPresent        command:          - sh          - "-c"          - |            set -ex            rm -rfv /mattermost/config/lost+found            cp /tmp/config/config.json /mattermost/config/config.json        volumeMounts:        - mountPath: /tmp/config/config.json          name: mattermost-init-config-json          subPath: config.json        - mountPath: /mattermost/config/          name: mattermost-config        - mountPath: /tmp/onelogin/          name: mattermost-onelogin      - name: init-plugins-config        image: busybox        imagePullPolicy: IfNotPresent        command:          - sh          - "-c"          - |            cp /mnt/plugins/init-plugins.sh /tmp && cd /tmp && chmod +x init-plugins.sh            ls -la            ./init-plugins.sh            ls -la /mattermost/plugins        volumeMounts:        - name: mattermost-init-plugins          mountPath: /mnt/plugins/        - name: mattermost-plugins          mountPath: /mattermost/plugins/        - name: mattermost-plugins-client          mountPath: /mattermost/client/plugins/      containers:      - name: mattermost-community-daily-app        image: "mattermost/mattermost-enterprise-edition:5.5.0"        imagePullPolicy: Always        terminationMessagePolicy: "FallbackToLogsOnError"        ports:        - containerPort: 8000          name: api        - containerPort: 8067          name: metrics        - containerPort: 8075          name: cluster        - containerPort: 8074          name: gossip        livenessProbe:          initialDelaySeconds: 90          timeoutSeconds: 5          periodSeconds: 15          httpGet:            path: /api/v4/system/ping            port: 8000        readinessProbe:          initialDelaySeconds: 15          timeoutSeconds: 5          periodSeconds: 15          httpGet:            path: /api/v4/system/ping            port: 8000        volumeMounts:        - mountPath: /mattermost/plugins/          name: mattermost-plugins        - mountPath: /mattermost/client/plugins/          name: mattermost-plugins-client        - mountPath: /mattermost/config/          name: mattermost-config      volumes:      - name: mattermost-init-config-json        configMap:          name: mattermost-community-daily-init-config-json          items:          - key: config.json            path: config.json      - name: mattermost-init-plugins        configMap:          name: mattermost-community-daily-init-plugins      - name: mattermost-onelogin        secret:          secretName: mattermost-community-daily-secret-onelogin      - name: mattermost-plugins        emptyDir: {}      - name: mattermost-plugins-client        emptyDir: {}      - name: mattermost-config        emptyDir: {}  ``` `

JobServer is a service that indexes Elasticsearch, syncs events, and more. Our JobServer deployment looks like this:

` apiVersion: extensions/v1beta1 kind: Deployment metadata:  name: mattermost-community-daily-jobserver  labels:    app: mattermost-community-daily-jobserver spec:  replicas: 1  template:    metadata:      labels:        app: mattermost-community-daily-jobserver        component: "jobserver"    spec:      initContainers:      - name: "init-mattermost-app"        image: "appropriate/curl:latest"        imagePullPolicy: "IfNotPresent"        command: ["sh", "-c", "until curl --max-time 5 https://mattermost-community.community:8000/api/v4/system/ping ; do echo waiting for Mattermost App come up; sleep 5; done; echo init-mattermost-app finished"]      containers:      - name: mattermost-community-daily-jobserver        image: "mattermost/mattermost-enterprise-edition:5.5.0"        imagePullPolicy: Always        command: ["mattermost", "jobserver"]        volumeMounts:        - mountPath: /mattermost/config/config.json          name: config-json          subPath: config.json      volumes:      - name: config-json        configMap:          name: mattermost-community-daily-init-config-json          items:          - key: config.json            path: config.json ``` `

And the services manifest is as follows:

` apiVersion: v1 kind: Service metadata:  name: mattermost-community-daily  labels:    app: mattermost-community-daily spec:  selector:    app: mattermost-community-daily  type: ClusterIP  ports:  - port: 8000    targetPort: 8000    protocol: TCP    name: mattermost-community-daily  - port: 8067    targetPort: 8067    protocol: TCP    name: mattermost-community-daily-app-metrics ``` `

The ConfigMap, which holds the Mattermost config, can be customized to your needs (additional setting configurations can be found here):

` apiVersion: v1 kind: ConfigMap metadata:  name: mattermost-community-daily-init-config-json  labels:    app: mattermost-community-daily data:  config.json: |            "ServiceSettings": {            ....                ...     ``` `

To set up plugins, we added an InitContainer, as you can see in the StatefulSet above.

Here’s the ConfigMap that holds the plugin definitions and the script required to download and configure the plugins:

` apiVersion: v1 kind: ConfigMap metadata:  name: mattermost-community-daily-init-plugins  labels:    app: mattermost-community-daily data:  init-plugins.sh: |    #!/bin/sh    PLUGINS_TAR="hovercardexample.tar.gz"    PLUGINS_TAR="${PLUGINS_TAR} jira2.tar.gz"    PLUGINS_TAR="${PLUGINS_TAR} memes.tar.gz"    PLUGINS_TAR="${PLUGINS_TAR} mattermost-github-plugin-linux-amd64.tar.gz"    PLUGINS_TAR="${PLUGINS_TAR} mattermost-plugin-autolink-linux-amd64.tar.gz"    PLUGINS_TAR="${PLUGINS_TAR} mattermost-zoom-plugin-linux-amd64.tar.gz"    PLUGINS_TAR="${PLUGINS_TAR} mattermost-jira-plugin-linux-amd64.tar.gz"    PLUGINS_TAR="${PLUGINS_TAR} mattermost-plugin-autotranslate-linux-amd64.tar.gz"    PLUGINS_TAR="${PLUGINS_TAR} com.github.matterpoll.matterpoll.tar.gz"    PLUGINS_TAR="${PLUGINS_TAR} com.mattermost.welcomebot.tar.gz"     for plugin_tar in ${PLUGINS_TAR};    do      wget https://mattermost-public-plugins-kubernetes.s3-website-us-east-1.amazonaws.com/${plugin_tar} -P /mattermost/plugins      cd /mattermost/plugins      tar -xzvf ${plugin_tar}      rm -f ${plugin_tar}    done ``` `

Learn more about how to migrate Mattermost to Kubernetes

Moving forward, we plan to make some additional changes to Mattermost to make it even easier to deploy and manage.

Stay tuned for updates—or, better yet, join the Kubernetes channel on our community server and take part in the discussion. You can find me as @cpanato there.

(Editor’s note: This post was written by Carlos Panato, Staff Software Engineer at Mattermost, Inc. If you have any feedback or questions about How to migrate Mattermost to Kubernetes, please let us know.)