Introduction to ChatOps
You’ve likely heard the term ChatOps. It’s a popular topic, and there are many commercial solutions out there that support it. But what is ChatOps exactly, and why should you look into it further?
ChatOps is a collaborative, conversation-centric way of working that brings people, discussions, bots, tools, and files together into one central location. It’s more of a methodology than a specific technology; a collective approach to running most DevOps workflows and building a collaborative team culture. That said, it helps to have a powerful tool at the center of your ChatOps practice, and Mattermost is an ideal fit.
Mattermost is an open source, self-hosted messaging platform for high-trust organizations, designed to enable workstream collaboration for DevOps and other enterprise teams. The platform gives you tools to automate processes, integrate systems, and build customized workflows—in essence, deliver the full value of ChatOps.
Benefits of ChatOps
- Gathers all the required people in one place, whether on-site or remote
- Provides a shared console or command line for easy collaboration
- Shortens the feedback loop and enables faster response times
- Reduces the mean time to restore service after system failure
- Captures the history of team activity, system events, problems, and solutions
If you are looking to explore the potential of ChatOps for your organization, then this guide is for you. We’ll walk through the progressive steps toward developing a powerful ChatOps practice on Mattermost. At each step, we’ll give you a peek under the hood at Mattermost and show you examples from our own ChatOps practice. You may need only a few of these steps, or you may want to pursue them all. Regardless, your teams will work better together and your organization will benefit from increased efficiency and productivity.
Step 1: Start small, do it manually for a bit
If yours is already a chat-based organization, then you’re a step ahead on the road to ChatOps. If not, then start by first developing a team culture and collaboration practice around messaging. Invite all your people to join a team channel on Mattermost and get the conversation started. Encourage them to post screenshots or code snippets. Think of it like a cooperative board game—just getting everyone gathered into the channel, discussing timely topics, and solving problems together is half the battle. Soon, your entire team will be making progress together daily as collaboration becomes easier and more fluid.
Developing a ChatOps practice also augments your team’s collective expertise. Instead of relying on communication siloes like email or meetings, everyone brings their own esoteric knowledge into the team channel, making it easy to share and learn from each other. Mattermost allows members to search message histories and catch up on missed conversations or find answers to a particular question.
Pro Tip > Don’t be afraid to over-add people to your channel in the beginning. As a rule, it’s always better to have more eyeballs in the channel and people will self-filter if the conversation is not relevant to them.
Example: Manual file sharing
Here, our team is solving a problem together using file sharing. Patni posts a screenshot, but Sudheer requests a log file instead. Mattermost’s rich messaging system gives them the flexibility to format information in a way that’s most valuable to the team.
Step 2: Automate the simple things
Once your team is comfortable with centralizing collaboration in the team channel, it’s time to branch out. Channel-based communication is key to many DevOps workflows, such as incident response or code deployment. Consider creating separate channels around specific topics—security, performance, build system, war rooms—whatever makes sense for your organization. Your goal is to set the stage for more powerful automation as your ChatOps practice matures.
As you think about automation, remember that you don’t need to start with a super powerful bot that will solve every problem. Start by automating your most basic alerts and making sure they show up in the relevant channel. Mattermost supports both incoming and outgoing webhooks. You can use incoming webhooks to make sure that the right alerts get to the right people. By automating a few things, you’ll get your first taste of the power and potential of ChatOps.
Pro Tip > Notice how your organization responds. When you deliver alerts to a topical channel that everyone can access, what happens to response times? Is collaboration faster, the workflow more efficient? This will help you look for ways to reinforce new communication habits and workflows using Mattermost.
Example: Solving a problem from a simple webhook
An alert comes in related to an issue on our corporate LDAP server. Our team is monitoring the channel, but Martin, the person who can handle the issue, is not in the channel. Carlos easily pings Martin to bring him into the channel to help solve the problem.
Step 3: Automate the harder things
By now you’re probably itching for more opportunities to automate your DevOps workflows. Beyond alerts, what else can you automate on Mattermost? Slash commands are a great way to build in automation quickly. Slash commands also allow you to easily integrate external applications into the Mattermost server. They function similarly to outgoing webhooks, except they can be used in any channel, including private channels and direct messages.
Each Mattermost instance comes with a set of built-in slash commands that are ready to use. Some automate tasks like sending out emails or managing users, and others provide shortcuts to common platform actions. Mattermost also enables your teams to build custom slash commands that integrate with your other systems, allowing you to trigger actions on external systems. They trigger an HTTP request to a web service that can in turn post one or more messages in response.
In addition to slash commands, Mattermost allows you to use richly formatted data, such as Markdown formatting, file attachments for logs, and syntax highlighting for code snippets. This saves your team time by allowing them to share and collaborate directly within the channel, rather than having to pull up the information in separate services or documents. The more rich information you can bring into the messaging experience, the faster people can understand context and details to be more productive.
Pro Tip > Consider how you share metrics data. Do your DevOps or SRE teams have access to reports that your engineers do not? Not everyone may be on the right email threads or connected to the right systems. Try using slash commands to bring those reports, richly formatted, into a public channel on Mattermost, so that the broader organization may benefit.
Example: Posting load test results using a slash command
A team member issued a slash command from within a channel. That command triggered our CI/CD pipeline to spin up a load test, and posted the results back into the channel for the team to review. The test data is richly formatted in a table for easy consumption.
Example: Rich messages and log files
Corey uploads a set of Nginx logs to a channel. It’s a text file, but when users click it, they can see the logs in place and start scrolling through them. This gives the team immediate access to deeper information about an incident or bug from within the channel, so they can better collaborate and take faster action.
Step 4: Start automating command and control
With alerts and rich data under your belt, it’s time to focus on automating your command and control. DevOps teams regularly get hit with common requests, like provisioning environments or running reports. It takes time to fulfill those requests—time that may be better spent on more important tasks. Mattermost goes beyond simply providing a central channel for initiating requests. It enables you to allow team members to fulfill those requests themselves. Using slash commands or outgoing webhooks, you can set up self-provisioning through the common messaging interface and allow anyone in your organization to query your systems and retrieve the data they need.
What’s more, self-provisioning happens in a public channel. This allows others to see what queries are being run and what data is being returned, driving broader information sharing and greater transparency within your organization. It also allows others to offer support if needed. The key here is that automation keeps people from making mistakes when running robust and potent queries that may otherwise harm your infrastructure. In addition, it allows non-developers like QA or support staff to run the commands themselves. By automating command and control, you empower more people to be self-sufficient.
Pro Tip > Try using slash commands to automate common command and control tasks. For example, you may want to allow QA teams to set up their own environments for ad hoc testing or load testing, or you may want to help product managers query analytics systems. Try automating deploys to production. The resulting message history will help the team learn from what went well or what went wrong.
Example: Using slash commands to cut a release build
We use a slash command to control our build system. It cuts a release build and posts the details into the appropriate channel, so the testers and others are informed and can respond accordingly. For Jenkins users, Mattermost’s Jenkins plugin allows you to initiate or cancel builds, pull logs, display build artifacts, and more directly from a chat window.
Step 5: Bring in the bots
Now, take all that great work you’ve done in previous steps and start converting it to a more human interface. Most people have a hard time remembering many slash commands and all their parameters. Why not move that heavy lifting over to a bot? Bots can use Mattermost’s REST API to integrate with Mattermost. Running a bot gives users a friendly interface and makes it easier to bring automation to a wider population of users. Bots allow people to bypass slash commands and use natural language instead. They also provide a way for integrations or plugins to privately message individuals instead of posting to public channels, which is useful for sending notifications that are relevant only to individual team members.
There are many great existing bot interfaces to choose from. Hubot is a good one to start with and Hubot plugins are available to make integration with Mattermost easier. You can use Hubot to connect to a range of DevOps systems and bring information into a relevant channel on Mattermost. For example, connect to Jenkins to retrieve server logs and results for various test servers in your QA channel, or connect to Grafana to display data visualizations in your analytics channel. Ultimately, you are upleveling the ChatOps experience for your organization at large by making workflows as easy and efficient as they can be.
Pro Tip > Your bot humanizes ChatOps for your organization, so give it some character. Make it fun! Your bot can be angry, ecstatic, goofy, nerdy—whatever you think will engage your users most and drive ongoing use. You can even build a story around your bot and integrate it into your team culture. “The angry bot did it, not me!” Blame the bot, not people.
Example: Hubot triggers a deployment, and a smile
A simple way we use Hubot is to make shipping code a lot more fun.
Step 6: Build smarter bots and workflows with plugins
At this point, your ChatOps practice has become pretty sophisticated—but there’s much more room for growth. One of the most powerful features of Mattermost is the ability to extend the platform and build complex, customized workflows on the platform. Plugins go beyond slash commands and enable you to more deeply integrate functionality from external systems directly into a channel or the Mattermost user interface. Your plugin-powered bot can drive many common DevOps tasks in a shared workspace where the team is most active, helping everyone collaborate more easily and stay productive.
Mattermost’s plugin framework gives you a simplified path to embedding specific applications into your team channels, or vice versa, embedding chat into those applications. You can leverage plugins on both the front end and back end. Web app plugins allow you to customize the Mattermost UI by overriding components, such as the profile popover, channel header, or sidebar. Over 600 integrations make it easier to integrate with third-party systems like Jira, Jenkins, Bitbucket, GitLab, and PagerDuty.
Pro Tip > If your team uses GitHub, try Mattermost’s GitHub’s plugin to connect your GitHub and Mattermost accounts. The plugin listens for incoming GitHub event webhooks or changes to repositories, so you can trigger actions in GitHub directly from Mattermost, get daily reminders, notifications about pull requests, mentions and more. See our examples below for inspiration.
Example: GitHub integration
Our GitHub bot takes control of the sidebar UI and enables users to directly access GitHub information and functionality from Mattermost. For example, users can see unread mentions and open pull requests, so they can act on them quickly.
Example: Jira integration
Our Jira bot enables a multi-modal experience in a messaging thread with rich popups that allow actions specific to the conversation. In this case, users can create a Jira ticket directly from the message itself. Users can also attach a Mattermost conversation to a Jira issue for easy reference in that system.
Step 7: Reach for Nirvana—bots talking to bots
Your ChatOps practice is humming along nicely on Mattermost thanks to rich integrations, smart bots, and a vibrant messaging-based team culture. Your people have learned to collaborate more efficiently, so why not your bots? Closing the loop between bots and humans will add that final layer of agility that will help your organization realize its full potential. We call it “Bot Nirvana.”
At Mattermost, we see some of our customers generating more than a million daily messages, 50% of which come from bots in their infrastructure. That’s 500k+ messages a day just from bots—an astonishing amount of activity. What are those bots doing exactly? One common use case is the war room. During an outage, a war room has so many messages flowing in, more than 20 per second, that a human can not actually read them. Instead, they have bots watching the war room, pulling out certain messages, and duplicating them in channels where people can easily interact with them. The bots in those channels then push messages from team members back to the bots in the war room channel, almost like a message bus. The key is that humans and bots are interacting using natural human language.
Bot collaboration has great potential to improve many more workflows on Mattermost. Bots talking to bots, bots talking to humans and visa versa—this is a state of Nirvana that can truly amplify organizational performance.
Pro Tip > Look for use cases in your organization where one or more bots can step in and help humans do their job better. It may be watching a message stream, such as in an active war room, performing rapid actions, or handling massive amounts of data. Empower both your people and your bots to work better together.
Use Mattermost’s GitHub plugin to post relevant links to GitHub in Mattermost, then use our Autolink plugin to convert those links into a more human readable format. Rather than show the full URL, you can use Autolink to automatically format the links to show repository names and pull request numbers.
Current Mattermost users can dive right into the recommendations outlined in this guide without anything further needed. If you’re new to Mattermost, it’s easy to get started on the platform. Mattermost’s open source version is self-hosted and free to use. If you’re interested in enterprise features such as integration with Active Directory/LDAP or SAML, you can download a free trial of our Enterprise Edition and run a pilot with your team for a month. Why not follow the suggestions in this guide to start sowing the seeds of your future ChatOps practice during your trial?
If you have questions about setting up ChatOps on the platform, we’re here to help. Sign up for a meeting with one of our Solution Architects, who work exclusively with our enterprise customers.
Mattermost is the open source collaboration platform built for development teams to drive innovation. Its on-premise and private cloud deployment provides the benefits of modern communication without sacrificing privacy. Mattermost gives enterprises the autonomy and extensibility they need to be more productive while meeting the requirements of IT and security teams.