Streamlining developer access to Prometheus and Grafana

Our Makefile entry point for developing against the Mattermost Server already tries to simplify things for developers as much as possible.

For example, when invoking make run-server, this build tooling takes care of all of the following (among other things!):

  • Downloading a suitable version of mmctl for API-driven command line interaction.
  • Checking your installed Go version to avoid cryptic error messages.
  • Downloading and starting various Docker containers:
    • mysql and postgres, both for a backend to the server and also to automate tests.
    • inbucket to simplify email testing without routing outside your machine.
    • minio to expose an S3-compatible interface to your local disk for hosting uploaded files
    • openldap to simplify testing alternate modes of authentication (requires access to the enterprise source).
    • elasticsearch to expose an improved search backend for posts and channels (requires access to the Enterprise source code).
  • Linking your server to an automatically-detected mattermost-webapp directory.
  • Optionally starting the server in the foreground if RUN_SERVER_IN_BACKGROUND is false.
  • Piping the structured server logs through a logrus decorator for easier reading.
  • Invoking the go compiler to run the server code in your local repository.

Adding Prometheus and Grafana

As of https://github.com/mattermost/mattermost-server/pull/16649, this build tooling now supports automating the setup of Prometheus and Grafana Docker containers. Together, these tools form the backbone of our Performance Monitoring (E20) functionality, scraping metrics from Mattermost to help give enterprise customers insight into the performance of their clusters. Automating this setup makes it easier for developers to test in a production-like environment, to help extend the monitoring available to customers, and even to run our certain kinds of load tests using our load testing framework.

Since the corresponding Mattermost functionality is only available with access to the Enterprise source code, these containers aren’t enabled by default. To turn them on, export the following environment variable into your shell:

export ENABLED_DOCKER_SERVICES="mysql postgres inbucket prometheus grafana"

Alternatively, you can follow the instructions in config.mk. Note that it’s not necessary to specify minioopenldap, or elasticsearch – these are added automatically when an Enterprise-enabled repository is detected.

The build tooling does more than just spin up two new containers, however. Various configuration files also automate the following:

This means that simply running make run-server gets you immediate access to the Mattermost Performance dashboards at http://localhost:3000:

Grafana dashboard

Prometheus is also available at http://localhost:9090:

Prometheus

Enabling metrics

Although the tooling is all set up to scrape metrics and display the dashboards, you’ll need to do some one-time work to enable metrics within the Mattermost server.

First, be sure you have an Enterprise license installed. Staff members should have access to the shared developer license, but you can also request a trial license in-product. Browse to /admin_console/about/license on your local Mattermost instance to set up the license.

Second, enable performance monitoring. You can do this manually via config.json and setting MetricsSettings.Enable to true, or by exporting MM_METRICSSETTINGS_ENABLE=true into your shell before starting the server, or by enabling this manually via the System Console at /admin_console/environment/performance_monitoring:

Performance monitoring

What’s next?

These changes weren’t just made in isolation. In fact, this is all just infrastructure work for another recent project to improve Mattermost performance monitoring. Stay tuned for a follow-up blog post!

Share this article:

mm

Jesse Hallam

Jesse Hallam is a Senior Lead Software Design Engineer at Mattermost, Inc. Prior to joining Mattermost, Jesse built real-time data pipelines at Greenfield Labs and a digital movie store at Kaleidescape. He is an alumnus of the University of Waterloo and now lives in Prince Edward Island.

To get future blog posts to your inbox, subscribe below.

We use cookies for advertising, social media and analytics purposes. Read about how we use cookies here. By continuing to use this site, you consent to our use of cookies.