Mattermost Platform

How to migrate Mattermost to Kubernetes

January 18, 2019

Platform

At Mattermost, we aim to develop a product that is simple to deploy and manage.

To this end, we recently examined Kubernetes, today’s most popular orchestration platform that uses containers. 

After determining the platform would make it even easier to deploy, manage and scale our product, we decided to migrate our “virtual office”—our Mattermost instance where we interact with customers and members of our community—from AWS to Kubernetes. That way, we’d be able to see what improvements we needed to make in order to make our platform more cloud-native.

Here’s how we did the migration:

Let’s explore each step in detail.

1. Decide where to run Kubernetes

First, we installed Kubernetes on the latest version of Mattermost.

Next, we needed to figure out where to run Kubernetes. We had several options, including on-premises, AWS, Azure, GCP or DigitalOcean.

We decided to go with AWS EKS because we currently don’t have the overhead to manage the entire Kubernetes cluster, only the worker nodes; AWS manages the master nodes.

(For the purposes of this post, we won’t focus on how to deploy Kubernetes on other platforms. If you need help with that, check this out.)

2. Set up and configure Ingress

After we deployed our Kubernetes cluster, we installed NGINX Ingress using this documentation.

To reduce WebSocket errors, we added cache and keep-alive in our ConfigMap:

    
kind: ConfigMap
apiVersion: v1
metadata:
 name: nginx-configuration
 namespace: ingress-nginx
 labels:
   app.kubernetes.io/name: ingress-nginx
   app.kubernetes.io/part-of: ingress-nginx
data:
 use-proxy-protocol: "true"
 http-snippet: "proxy_cache_path /cache/mattermost levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;"
 keep-alive: "3600"

The Kubernetes service will look like this:

    
kind: Service
apiVersion: v1
metadata:
 name: ingress-nginx
 namespace: ingress-nginx
 labels:
   app.kubernetes.io/name: ingress-nginx
   app.kubernetes.io/part-of: ingress-nginx
 annotations:
   service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
   service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
spec:
 type: LoadBalancer
 selector:
   app.kubernetes.io/name: ingress-nginx
   app.kubernetes.io/part-of: ingress-nginx
 ports:
   - name: http
     port: 80
     targetPort: http
   - name: https
     port: 443
     targetPort: https

We also needed to add volume to store the cache to the deployment. So we updated the deployment manifest:

    
...
     volumeMounts:
     - mountPath: /cache/mattermost
       name: mattermost-cache
...
     volumes:
     - emptyDir: {}
       name: mattermost-cache

To get the TLS certificates—which are not required but are recommended for security reasons—we installed a cert manager using the helm chart. For more information, follow the instructions here.

3. Set up the database

In our legacy deployment (i.e., running servers on EC2), we had our database set up in AWS using RDS, a managed service by AWS.

While we transitioned from EC2 to Kubernetes, we decided to keep using RDS for our database. We do, however, plan to move the database to Kubernetes as well. It’s only a matter of time.

Here, we had to make a few configuration changes to the security groups and VPCs in order to allow the Kubernetes cluster to talk to the database.

4. Set up Mattermost

To set up Mattermost Enterprise Edition, we first had to set up a deployment manifest. (If you use Team Edition, use the official Mattermost helm chart for help with installation.)

Here’s what our manifest looks like:

    
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mattermost-community-daily-app
 labels:
   app.kubernetes.io/name: mattermost-community-daily-app
   app.kubernetes.io/component: "mattermost-community-daily-app"
spec:
 replicas: 2
 strategy:
   type: RollingUpdate
   rollingUpdate:
     maxUnavailable: 1
 selector:
   matchLabels:
     app.kubernetes.io/name: mattermost-community-daily-app
     app.kubernetes.io/component: "mattermost-community-daily-app"
 template:
   metadata:
     labels:
       app.kubernetes.io/name: mattermost-community-daily-app
       app.kubernetes.io/component: "mattermost-community-daily-app"
     annotations:
       prometheus.io/scrape: "true"
       prometheus.io/port: "8067"
       prometheus.io/path: "/metrics"
   spec:
     initContainers:
     - name: init-config
       image: busybox
       imagePullPolicy: IfNotPresent
       command:
         - sh
         - "-c"
         - |
           set -ex
           rm -rfv /mattermost/config/lost+found
           cp /tmp/config/config.json /mattermost/config/config.json
       volumeMounts:
       - mountPath: /tmp/config/config.json
         name: mattermost-init-config-json
         subPath: config.json
       - mountPath: /mattermost/config/
         name: mattermost-config
       - mountPath: /tmp/onelogin/
         name: mattermost-onelogin
     - name: init-plugins-config
       image: busybox
       imagePullPolicy: IfNotPresent
       command:
         - sh
         - "-c"
         - |
           cp /mnt/plugins/init-plugins.sh /tmp && cd /tmp && chmod +x init-plugins.sh
           ls -la
           ./init-plugins.sh
           ls -la /mattermost/plugins
       volumeMounts:
       - name: mattermost-init-plugins
         mountPath: /mnt/plugins/
       - name: mattermost-plugins
         mountPath: /mattermost/plugins/
       - name: mattermost-plugins-client
         mountPath: /mattermost/client/plugins/
     containers:
     - name: mattermost-community-daily-app
       image: "mattermost/mattermost-enterprise-edition:5.5.0"
       imagePullPolicy: Always
       terminationMessagePolicy: "FallbackToLogsOnError"
       ports:
       - containerPort: 8000
         name: api
       - containerPort: 8067
         name: metrics
       - containerPort: 8075
         name: cluster
       - containerPort: 8074
         name: gossip
       livenessProbe:
         initialDelaySeconds: 90
         timeoutSeconds: 5
         periodSeconds: 15
         httpGet:
           path: /api/v4/system/ping
           port: 8000
       readinessProbe:
         initialDelaySeconds: 15
         timeoutSeconds: 5
         periodSeconds: 15
         httpGet:
           path: /api/v4/system/ping
           port: 8000
       volumeMounts:
       - mountPath: /mattermost/plugins/
         name: mattermost-plugins
       - mountPath: /mattermost/client/plugins/
         name: mattermost-plugins-client
       - mountPath: /mattermost/config/
         name: mattermost-config
     volumes:
     - name: mattermost-init-config-json
       configMap:
         name: mattermost-community-daily-init-config-json
         items:
         - key: config.json
           path: config.json
     - name: mattermost-init-plugins
       configMap:
         name: mattermost-community-daily-init-plugins
     - name: mattermost-onelogin
       secret:
         secretName: mattermost-community-daily-secret-onelogin
     - name: mattermost-plugins
       emptyDir: {}
     - name: mattermost-plugins-client
       emptyDir: {}
     - name: mattermost-config
       emptyDir: {}

```

JobServer is a service that indexes Elasticsearch, syncs events, and more. Our JobServer deployment looks like this:

    
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 name: mattermost-community-daily-jobserver
 labels:
   app: mattermost-community-daily-jobserver
spec:
 replicas: 1
 template:
   metadata:
     labels:
       app: mattermost-community-daily-jobserver
       component: "jobserver"
   spec:
     initContainers:
     - name: "init-mattermost-app"
       image: "appropriate/curl:latest"
       imagePullPolicy: "IfNotPresent"
       command: ["sh", "-c", "until curl --max-time 5 http://mattermost-community.community:8000/api/v4/system/ping ; do echo waiting for Mattermost App come up; sleep 5; done; echo init-mattermost-app finished"]
     containers:
     - name: mattermost-community-daily-jobserver
       image: "mattermost/mattermost-enterprise-edition:5.5.0"
       imagePullPolicy: Always
       command: ["mattermost", "jobserver"]
       volumeMounts:
       - mountPath: /mattermost/config/config.json
         name: config-json
         subPath: config.json
     volumes:
     - name: config-json
       configMap:
         name: mattermost-community-daily-init-config-json
         items:
         - key: config.json
           path: config.json
```

And the services manifest is as follows:

    
apiVersion: v1
kind: Service
metadata:
 name: mattermost-community-daily
 labels:
   app: mattermost-community-daily
spec:
 selector:
   app: mattermost-community-daily
 type: ClusterIP
 ports:
 - port: 8000
   targetPort: 8000
   protocol: TCP
   name: mattermost-community-daily
 - port: 8067
   targetPort: 8067
   protocol: TCP
   name: mattermost-community-daily-app-metrics
```

The ConfigMap, which holds the Mattermost config, can be customized to your needs (additional setting configurations can be found here):

    
     apiVersion: v1
kind: ConfigMap
metadata:
 name: mattermost-community-daily-init-config-json
 labels:
   app: mattermost-community-daily
data:
 config.json: |
   
       "ServiceSettings": {
           ....
       
       ...
   
```

To set up plugins, we added an InitContainer, as you can see in the StatefulSet above.

Here’s the ConfigMap that holds the plugin definitions and the script required to download and configure the plugins:

    
apiVersion: v1
kind: ConfigMap
metadata:
 name: mattermost-community-daily-init-plugins
 labels:
   app: mattermost-community-daily
data:
 init-plugins.sh: |
   #!/bin/sh
   PLUGINS_TAR="hovercardexample.tar.gz"
   PLUGINS_TAR="${PLUGINS_TAR} jira2.tar.gz"
   PLUGINS_TAR="${PLUGINS_TAR} memes.tar.gz"
   PLUGINS_TAR="${PLUGINS_TAR} mattermost-github-plugin-linux-amd64.tar.gz"
   PLUGINS_TAR="${PLUGINS_TAR} mattermost-plugin-autolink-linux-amd64.tar.gz"
   PLUGINS_TAR="${PLUGINS_TAR} mattermost-zoom-plugin-linux-amd64.tar.gz"
   PLUGINS_TAR="${PLUGINS_TAR} mattermost-jira-plugin-linux-amd64.tar.gz"
   PLUGINS_TAR="${PLUGINS_TAR} mattermost-plugin-autotranslate-linux-amd64.tar.gz"
   PLUGINS_TAR="${PLUGINS_TAR} com.github.matterpoll.matterpoll.tar.gz"
   PLUGINS_TAR="${PLUGINS_TAR} com.mattermost.welcomebot.tar.gz"

   for plugin_tar in ${PLUGINS_TAR};
   do
     wget http://mattermost-public-plugins-kubernetes.s3-website-us-east-1.amazonaws.com/${plugin_tar} -P /mattermost/plugins
     cd /mattermost/plugins
     tar -xzvf ${plugin_tar}
     rm -f ${plugin_tar}
   done
```

Learn more about how to migrate Mattermost to Kubernetes

Moving forward, we plan to make some additional changes to Mattermost to make it even easier to deploy and manage.

Stay tuned for updates—or, better yet, join the Kubernetes channel on our community server and take part in the discussion. You can find me as @cpanato there.

(Editor’s note: This post was written by Carlos Panato, Staff Software Engineer at Mattermost, Inc. If you have any feedback or questions about How to migrate Mattermost to Kubernetes, please let us know.)

Read more about:

Kubernetes
mm

Carlos Panato is a Staff Software Engineer at Mattermost, Inc. who’s working on development and infrastructure using Kubernetes and containers. Previously, he’s worked on development, testing, processes and management. Prior to joining Mattermost, Carlos held several software engineering roles at companies like Dell, CoreOS, Meltwater, HERE and Chaordic Systems. He graduated with a degree in mechatronic engineering from Pontifícia Universidade Católica do Rio Grande do Sul.

Read More Platform Articles