Mattermost as a communication gateway

The Mattermost platform is a powerful messaging tool that enables secure team collaboration. Rather than creating unique Mattermost plugins for each tool, Cognitio created a custom SMTP mail intercept capability that leverages AWS Lambda functionality and custom code to create integrations for a number of third-party applications.

This post is about the basic functionality we created for a customer that joined Confluent, Jira, and Splunk with Mattermost Enterprise Edition E20 to send and receive messages and tickets into custom Mattermost groups, with permission integrated via AD/LDAP.

Cognitio proposed to develop a solution that leveraged native SMTP functionality in the third-party tools to achieve the desired goals. By accepting incoming SMTP messages from each tool and intercepting the message prior to relay, these messages can be delivered to a queue for processing and posting to Mattermost.

The general flow of the Gateway application is depicted the following figure:

In general, notification emails are sent from third-party tools (e.g., Jira) using the tools’ native SMTP/SMTPS interface configuration. Each tool is configured to send emails through a postfix instance running on the server that the Gateway application is on. Rather than relaying the incoming messages, the postfix application is configured to send incoming email messages to the Gateway application via an intermediary processing script known as the mail_eater.

The mail_eater simply receives the incoming email message, converts it into a text file, and passes it into the Gateway S3 bucket where the Lambda function will trigger. The Gateway function handles the primary processing and transformation of the incoming message, processes attachments, performs the necessary LDAP queries and ultimately the posting of the message to Mattermost.

Process flow

There is one primary mode of operation for Gateway: parse.

The workflow in parse mode is as follows:

• Incoming email message is received by the MTA which relays the message to the configured file route
• Mail_eater reads the incoming message from standard in, creates a text file based on the incoming MIME message, and then uploads the file to the “incoming” folder of the S3 bucket.
• The Gateway Lambda function is triggered by the upload, calling the handler function. The handler gathers some basic information about the event and the name of the uploaded file, before passing that information to the main function.
• The local running configuration is parsed by the function and a configuration variable is created to be referred to by the application.
• The incoming message is parsed and decomposed for posting to Mattermost
  • Message contents are parsed and prepped for posting, including converting HTML to Markdown if necessary
  • Attachments from the original message are downloaded to the /tmp/ directory of the Lambda function, and the names of the attachments are stored in an array.
• Source and destination users are identified and validated
  • Source and destination email addresses from the message are looked up in LDAP to retrieve the associated user IDs
  • Destination users are validated against blacklist and whitelist groups in LDAP
  • A list of destination users who have been authorized to receive notifications for the source application is returned
• Lambda function posts message to Mattermost
  • The password for the source application is retrieved from the secrets file in S3
  • An authentication token is retrieved from Mattermost for the application user
  • The Mattermost ID for each user is retrieved from Mattermost
  • A direct message channel is created between the source application user and each destination user
  • If there are attachments, the attachments are uploaded to Mattermost and the corresponding file IDs are collected from Mattermost.
  • The message is posted to each of the created direct message channels. If there are attachments, the file IDs are included as part of the post request.

Solution details

Cognito wrote the solution using a series of Python scripts leveraged in an AWS instance that accessed an S3 bucket and Lambda to simplify the handling process and enforce security that was needed for the client. The solution will run on any hardware that supports running Python 3.7+.

The software for our solution was as follows:

General Software	Python Packages (included in package)
● Python 3.7+ ● Postfix 2.x ● Cyrus SASL 2.x ● Mattermost 5.12+ (API v4)	● Html2text 2019.9.26+ ● Ldap3 2.6.1+ ● Mail-parser 3.10.0+ ● Pybase64 1.0.1+ ● Requests 2.6+ ● Boto3 (for MTA only)

The first step was to set up a virtual server in AWS and then an S3 bucket. The S3 instance held the configuration files, secrets file (for passwords needed for the various third-party software), and copies of processed messages (which were expunged after processing). The folders set up for processing included incoming, quarantine (for when parsing failed), and processed.

We then set up a series of IAM policies/roles that were needed for execution in the AWS environment. After that, we set up a series of LDAP Users/Groups needed to receive and post messages as well as to validate that users were properly authorized to see incoming messages. This basically boiled down to LDAP user, App user, Blacklist group, and App groups.

Next, we deployed an EC2 instance to host the Message Transfer Agent (MTA) and set up necessary security and installed and configured Postfix from the native package manager.

The Gateway application stack is written entirely in Python and delivered as a set of versioned zip files, one with the MTA components of the stack and one with the Lambda function. In order to function, the MTA must have Python 3 installed, and the boto3 package must also be installed.

Deploying the Gateway MTA is as easy as unzipping the zip file into the intended operating directory and ensuring that the Python files are read accessible by the service account that is executing the Gateway workflow (i.e., mailsvc) set later during the postfix configuration. Gateway does not require a specific operating location. However, we created one for this solution.

The next steps were to install and configure SASL for authentication of various products and users. We then setup a Lambda trigger based on the trigger being “any creation event’”so that it would activate when any message was received.

On the Mattermost side, we needed to create a series of accounts (one per notification application) that would be authorized to post messages. Since our Gateway posts direct messages to notification recipients these users had to have the ability to post direct messages to all users. For example, an incoming message from Jira sent to [email protected] will be posted as a direct message from the Jira Mattermost service account to the Mattermost account associated with [email protected] based on an LDAP lookup.

Each application that will use this solution must be configured with an outbound mail server in order to send notifications. The actual configuration varies by third-party application and that a set of shared credentials be used to authenticate and send messages (this was done as part of the SASL install).

When the Gateway is parsing an incoming mail, it looks at the source email address (e.g., [email protected]) and removes the svc string to reveal the application name (Jira) which becomes the key that the Gateway uses to retrieve the LDAP group from one of the Python config files.

Final thoughts

This integration runs very smoothly at scale and can be configured to work with any number of third-party applications for which there is no native API and where the third-party application has the ability to send SMTP messages. We found Python to be the easiest solution in working with Lambda. Overall, it helps to unlock the power of Mattermost as a central communication hub for an enterprise.

If you’re interested in working with Cognitio on functionality like this, get in touch with us.