Leveraging AI for community moderation: A tale of two assistants
Ask any community member who’s ever donned the moderator tag in a new forum and they’ll agree that handling a multitude of new general questions, technical queries, and troubleshooting scenarios can be challenging at first.
However, thanks to the rise of AI-powered chatbots, there are a lot of new opportunities to find innovative ways to tackle these challenges.
The Mattermost Peer-to-Peer Forum is a good example of a community consisting primarily of developers. As such, a lot of technical help and troubleshooting goes on there.
As a new community moderator who’s admittedly not very familiar with software development, I decided to explore the possibilities of utilizing AI-powered chatbots for some help. My goals were clear: Streamline the troubleshooting process on tickets the community couldn’t immediately solve, save time otherwise spent sifting through docs, and overall enhance the experience for our community.
This journey led to the creation of two formidable contenders in the world of AI-assisted support: Matterbot 2.0 and Matterbot 3.0. We had a v1.0 at one point too, but that was basically just me telling ChatGPT to pretend to be a top-tier forum moderator for Mattermost and thinking its responses were cool.
With v2 and v3, I started to learn a bit more about leveraging large language models (LLMs). Each of the bots had their own unique strengths and abilities, and I was eager to put them to the test.
The birth of Matterbot 2.0
My AI exploration journey really began with Matterbot 2.0, a custom chatbot I built using Zapier’s Interfaces platform. I trained the bot using a variety of Mattermost help documents and FAQs. The aim was simple: to assist in crafting responses to the technical questions that weren’t immediately answered on the forums.
After initial testing, Matterbot 2.0 showed promise with a 38% success rate, but I knew there was still lots of room for improvement. I categorized a “win” as a response that led to a solution or a positive conversation and a “loss” as no response or a negative outcome.
The rise of Matterbot 3.0: The Mattermost AI assistant
Meanwhile, the Mattermost engineering team was working diligently on OpenOps AI projects, which culminated in the creation of the Mattermost AI Assistant.
I decided to put Matterbot 3.0 to the test using a demo version of the assistant, pitting it against Matterbot 2.0 in an A/B test. The results have been truly impressive:
Fueled by the Mattermost AI Assistant, Matterbot 3.0 demonstrated a remarkable 57% success rate, outperforming its predecessor by creating more positive dialogues and more efficient solutions.
The impact of AI on community moderation
This journey underscores the growing role of AI bots in the field of community management and customer support. As a Mattermost community moderator, I’ve come to appreciate the key advantages AI-powered assistants offer:
- Time efficiency. AI bots quickly deliver response drafts that serve as starting templates. As such, they save valuable time and allow me to concentrate on highlighting the best possible solutions.
- Inspiration and assistance. When I’m not sure where to start or I have a lot of docs to read through, the bots help jumpstart my thinking, offering valuable support in crafting well-informed replies.
- Workflow automation. The bots automate tedious tasks — from sifting through FAQs to suggesting routine troubleshooting responses — which makes community moderation more efficient.
The success of Matterbot 3.0, driven by the Mattermost AI Assistant, serves as an early proof-of-concept on the evolving capabilities of AI bots in customer support and community management. These powerful tools not only streamline the support process but also enhance the overall user experience within our community forums.
As AI technology continues to advance, we have a powerful ally at our disposal — one that not only saves time but also enriches the support experience for our users.
In my opinion, the future of community management is undeniably brighter with AI by our side.
Want to see the Mattermost AI Assistant in action?
Head on over to the Mattermost Peer-to-Peer Forum and search for any active troubleshooting threads that have responses from John Combs.
Alternatively, chat with the Mattermost AI Assistant yourself in a DM channel by installing the Mattermost AI Plugin yourself.
