testing OpenAPI with Schemathesis

OpenAPI testing with Schemathesis and Golang

September 27, 2022

Engineering

When you maintain an internal or public-facing API, the API documentation is an important component of the overall user/developer experience. One of the industry standards for such documentation is the OpenAPI specification. 

With an OpenAPI specification, you define a contract that specifies how your API should behave, but nothing stops the parties involved from breaking such a contract (e.g., using a wrong implementation or invalid input). API testing allows you to ensure conformance, validate that your documentation matches the actual API implementation, catch logical errors, and ensure that your code can safely handle invalid inputs.

Schemathesis is a tool for testing web applications based on their OpenAPI (or GraphQL) specifications. The actual implementation can be in any language as Schemathesis only needs a valid API specification. It uses a method known as property-based testing to provide high input variation for test inputs and tests easy-to-ignore corner cases. It can also use explicit examples written into the spec and lets you replay every request made during the tests via VCR cassettes.

In this article, we’ll explore API testing using Schemathesis and the Gin framework and see how Schemathesis helps us provide better guarantees about our API and its documentation. The demo application is a bucket list API with support for adding, reading, and deleting items from there.

Requirements

To follow along with this tutorial, you should:

  • Be familiar with the OpenAPI specification.
  • Have Python (and pip) installed because they are dependencies of the Schemathesis CLI. You can use the Schemathesis docker images (in that case, replace all instances of the schemathesis command in the article with docker run --network=host schemathesis/schemathesis:v3.15.6. Note that use of the Docker images require a valid Schemathesis.io token that can be obtained after registration.
  • Optionally have Go installed. The provided demo API is in Go, but Schemathesis is language-agnostic, so feel free to implement the API in your preferred language.

Getting started with Schemathesis

Install Schemathesis with pip as in the command below:

pip install schemathesis==v3.15.4

Next, create a new directory in your preferred location and enter the command below (I am naming mine schemathesis-demo):

mkdir -p ~/schemathesis-demo && cd ~/schemathesis-demo

The project directory created above will hold our OpenAPI specification and Go source files for our demo server.

Preparing our OpenAPI specification

To prepare the first iteration of our OpenAPI schema, let’s define the metadata and our endpoints. 
Create a new OpenAPI spec file named openapi.yaml  in the project directory and add the code below:

openapi: "3.0.0"
info:
  version: 0.1.0
  title: Bucket List API
  license:
    name: "MIT"
servers:
  - url: http://localhost:5000/api

paths:
  /items:
    get:
      summary: Get all items in the bucket list
      operationId: listItems
      tags:
        - items
      parameters:
        - name: year
          in: query
          description: Only return the bucket list for the specified year.
          required: false
          schema:
            type: string
            pattern: '^\d{4}$'
      responses:
        '200':
          description: A list of items in the bucket list
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Items"
    post:
      summary: Add a new item to the list
      operationId: addItem
      tags:
      - items
      responses:
        '201':
          description: Item added successfully
          content:
            application/json:
              schema: 
                $ref: "#/components/schemas/Item"
  
  /items/{itemId}:
    get:
      summary: Retrieve a bucket list item
      operationId: getItem
      tags:
      - items
      parameters:
        - name: itemId
          in: path
          required: true
          description: ID of the item to retrieve
          schema:
            type: integer
      responses:
        '200':
          description: Item retrieved successfully
          content:
            application/json:
              schema: 
                $ref: "#/components/schemas/Item"
    delete:
      summary: Remove an item from the bucket list
      operationId: deleteItem
      tags:
      - items
      parameters:
        - name: itemId
          in: path
          required: true
          description: ID of the item to retrieve
          schema:
            type: integer
      responses:
        '204':
          description: Item deleted successfully

The schema above defines four endpoints:

  1. GET /items: This returns the list of all items in the bucket list. It also allows for a “year” parameter which users can use to filter the list.
  2. POST /items: It allows users to add a new item to the list. The request body should match the Item component defined below.
  3. GET /items/{itemId: This allows users to retrieve a single item from the list using the item’s ID.
  4. DELETE /items/{itemId: This allows users to delete an existing item from the list using the item’s ID.

It also references three components:

  • Item represents a single bucket list item.
  • Items represents a collection (i.e., an array) of items.
  • Error represents the general structure of errors returned by our API.

In the same OpenAPI spec file (openapi.yaml), let’s define these components and their fields/properties as follows:

components:
  schemas:
    Item:
      type: object
      required:
      - title
      - description
      properties:
        id:
          type: integer
          description: ID of the bucket list item.
        title:
          type: string
          description: Title of the item.
        description:
          type: string
          description: More detailed description of the item.
        year:
          type: string
          description: Target year to have done this time.
          pattern: '^\d{4}'
    Items:
      type: object
      items:
        $ref: "#/components/schemas/Item"
    Error:
      type: object
      required:
        - message
      properties:
        message:
          type: string
        data:
          type: object

First steps with Schemathesis

For local development, Schemathesis requires a schema and the API base URL. With our schema in place, let’s also create a basic Go server which we will later extend to this server to serve the endpoints defined in the schema above.

Initialize the Go project using your details based on the following example:

go mod init gitlab.com/idoko/schemathesis-demo

The command above makes the project a valid Go module. You can check out the official Go blog to learn more about modules and how to use them.

We will use the Gin framework as a router for our API server; run the command below to add it as a project dependency:

go get -u github.com/gin-gonic/gin

Next, create a main.go file:

package main

import "github.com/gin-gonic/gin"

func main() {
        r := gin.Default()
        r.Run(":8080")
}

This imports the Gin router and opens a server on http://localhost:8080. Start the server by running the following command on your terminal:

go run ./main.go

We can get a feel of Schemathesis at this point by executing the test in a separate terminal using the following command:

schemathesis run --base-url=http://localhost:8080 openapi.yaml

The command runs the tests against the OpenAPI schema we defined earlier by making HTTP requests to the base URL we specified and produces an output similar to the one below:

Schemathesis success output following a basic test
Schemathesis success output following a basic test

While none of the endpoints is implemented yet, the output above reports Schemathesis as successful. This is because, by default, it only checks that the application doesn’t crash with an internal server (HTTP 500) error for different kinds of inputs. Schemathesis provides five different check types:

  1. not_a_server_error checks that the application doesn’t return a HTTP 5xx error for any endpoint (unless it is explicitly defined for such).
  2. status_code_conformance checks that the HTTP code returned by the application matches what is defined in the API schema.
  3. content_type_conformance checks that the HTTP content type returned by the application is defined in the API schema.
  4. response_headers_conformance checks that all headers present in the actual response from the application is defined in the API schema.
  5. response_schema_conformance checks that the body of each response matches what is defined in the API schema.

Let’s re-run the tests with all of these checks enabled using the command below:

schemathesis run --checks=all --base-url=http://127.0.0.1:8080 ./openapi.yaml

This time, the test fails with a summary similar to the screenshot below.

Schemathesis error output after enabling all checks
Schemathesis error output after enabling all checks

In the screenshot above, no endpoint passed the status_code_conformance test because our server currently returns a HTTP 404 error for all endpoints. Since we are yet to specify what a 404 error should look like in the schema, Schemathesis assumes that the response structure is correct and marks the other tests as passed.

Now that we have failing tests, let’s work to make them pass by providing an actual API implementation.

Implementing API endpoints with Gin

Jump to the Docker section if you would rather follow along with a ready-made API. 

For simplicity, we will store all the bucket list items in memory. This is achieved by maintaining a global list of items wrapped in an API structure that we will define. Open the main.go file created earlier and add the following code before the main function:

type Item struct {
        Id          string `json:"id,omitempty"`
        Title       string `json:"title"`
        Description string `json:"description"`
        Year        string `json:"year,omitempty"`      
}

type API struct {
   lock sync.Mutex
   items map[string]Item
}

The code above adds two new structs:

  • Item, which represents an individual bucket list item and contains the fields we want. It also directs Go to ignore the Id and Year fields, making them optional.
  • API, which maintains a list of items. To make operations like item lookup and deletions easier, this list is stored as a mapping of Item ID (integer) to the Item object.

Note: An implementation detail is that Go maps are not safe for concurrent use, hence we have introduced a mutex lock to protect it during modifications.

Next, we will implement the necessary route handlers for each of the defined endpoints.

Adding a new item

To create the endpoint for adding a new item, we will add a receiver function named createItem on our API struct. In main.go, add the function implementation below to your main.go file:

func (a *API) createItem(gc *gin.Context) {
   var itemReq Item
   if err := gc.ShouldBindJSON(itemReq); err != nil {
      gc.JSON(http.StatusBadRequest, gin.H{
         "message": "failed to parse request body",
      })
      return
   }

   itemReq.Id = uuid.New().String()

   a.lock.Lock()
   a.items[itemReq.Id] = itemReq
   a.lock.Unlock()

   gc.JSON(http.StatusCreated, gin.H{
      "message": "item created",
      "data": itemReq,
   })
   return
}

This function accepts a Gin context as parameter, making it qualified to be used as a route handler. It then parses the request body and generates a new UUID string which is then used as the item’s ID.

Getting an item

To retrieve a single item, we read the item ID from the request URL and fetch the corresponding item from the items map. We also perform some extra validations like making sure the ID is a valid UUID string and that such an item exists in the map.

func (a *API) getItem(gc *gin.Context) {
   itemId := gc.Param("itemId")
   if itemId == "" {
      gc.JSON(http.StatusBadRequest, gin.H{
         "message": "Empty or invalid item ID",
      })
      return
   }

   if _, err := uuid.Parse(itemId); err != nil {
      log.Printf("failed to parse item ID '%s': '%s'", itemId, err.Error())
      gc.JSON(http.StatusBadRequest, gin.H{
         "message": fmt.Sprintf("Failed to parse Item ID: '%s'", itemId),
      })
      return
   }

   var item Item
   var ok bool

   a.lock.Lock()
   defer a.lock.Unlock()
   if item, ok = a.items[itemId]; !ok {
      gc.JSON(http.StatusNotFound, gin.H{
         "message": fmt.Sprintf("No item found with ID '%s'", itemId),
      })
      return
   }
   gc.JSON(http.StatusOK, gin.H{
      "message": "successful",
      "data":    item,
   })
   return
}

Deleting an item

The implementation for our DELETE /item endpoint is quite similar to GET /item, except in this case, we remove the corresponding value from the map after validation and return an empty response:

func (a *API) deleteItem(gc *gin.Context) {
   itemId := gc.Param("itemId")
   if itemId == "" {
      gc.JSON(http.StatusBadRequest, gin.H{
         "message": "Empty or invalid item ID",
      })
      return
   }

   var item Item
   var ok bool

   a.lock.Lock()
   defer a.lock.Unlock()
   if item, ok = a.items[itemId]; !ok {
      gc.JSON(http.StatusNotFound, gin.H{
         "message": fmt.Sprintf("No item found with ID '%s'", itemId),
      })
      return
   }

   delete(a.items, item.Id)
   gc.JSON(http.StatusNoContent, nil)
   return
}

Retrieving all items

Next, we add the code for retrieving all items. This also comes with the ability to only show items slated for a specific year:

func (a *API) getItems(gc *gin.Context) {
   matches := make([]Item, len(a.items), 0)
   year := gc.Query("year")

   for _, item := range a.items {
      // if year is defined, we only pick items whose year matches,
      // otherwise, we pick all the items in the map
      if year == "" || item.Year == year {
         matches = append(matches, item)
      }
   }

   gc.JSON(http.StatusOK, gin.H{
      "message": "successful",
      "data":    matches,
   })
}

Registering route handlers

Next, update the main function to register the handlers we’ve defined as shown below:

func main() {
   r := gin.Default()

   a := API{
      items: map[string]Item{},
   }

   r.GET("/items/:itemId", a.getItem)
   r.DELETE("/items/:itemId", a.deleteItem)
   r.POST("/items", a.createItem)
   r.GET("/items", a.getItems)

   r.Run(":8080")
}

Finally, update the import block at the top of the file to bring in the new dependencies you need:

import (
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/google/uuid"
	"log"
	"net/http"
	"sync"
)

Now that the route handlers are all implemented, install the new dependencies and restart the Go server with the commands below:

go get github.com/google/uuid
go run ./main.go

Alternatively running the API server with Docker

The Go-based API is also available as a Docker image for you to use. The command below will run the prepared Docker image and expose port 8080 on your localhost for Schemathesis to use:

docker run -p 8080:8080 -t idoko/schemathesis-demo:latest

Debugging test errors

With the API up and running, you can run Schemathesis (with all checks enabled) against the API URL using the same command from earlier:

schemathesis run --checks=all --base-url=http://127.0.0.1:8080 ./openapi.yaml

Again, this fails with the summary below even though we’ve already implemented all the endpoints.

Schematheis error output after implementing endpoints

Looking closely at the output, though. We notice that the number of tests has dramatically increased (from 11 to 108). This is because as Schemathesis gets a success response, it further varies the inputs to cover more edge cases. We also notice an error similar to the one below:

This happens because our endpoints return HTTP responses (e.g., 404 for missing items) that we still haven’t defined in the API schema hence, our tests fail the status_code_conformance check again.


To get the test to pass, update each endpoint response with the possible errors it can return. As an example, here’s what the GET /items path looks like after the change:

...
paths:
  /items:
    get:
      summary: Get all items in the bucket list
      operationId: listItems
      tags:
        - items
      parameters:
        - name: year
          in: query
          description: Only return the bucket list for the specified year.
          required: false
          schema:
            type: string
            pattern: '^\d{4}$'
      responses:
        '200':
          description: A list of items in the bucket list
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                    description: Response summary
                  data:
                    $ref: "#/components/schemas/Items"
        '4XX':
          description: Item not found
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
        '5XX':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"
...

Re-run the tests with:

schemathesis run --checks=all --base-url=http://127.0.0.1:8080 ./openapi.yaml

This time, you should get an even more number of tests, all of which are successful as shown in the screenshot below:

Filtering Schematheis operations

During development, it is possible that we want to limit the API tests to only one endpoint. Schemathesis provides a couple of filtering options that allows you to filter endpoints by name, HTTP method, etc. For example, we can test only the items endpoint (GET /items and POST /items) by running:

schemathesis run --checks=all --endpoint="^/items$" --base-url=http://127.0.0.1:8080 ./openapi.yaml

The command above yields the following output:

Learn more about using Schemathesis

Testing your API during development and as part of your continuous integration process helps you provide better guarantees about the stability of the API, and Schemathesis eases that process for you. It also comes with extra niceties such as:

  • Stateful testing, which allows you to define the relationship between endpoints (e.g., creating a resource and finding it).
  • Test recording, which enables you to record HTTP requests and replay them for debugging, among other use cases.

To learn more about Schemathesis and how it works under the hood, explore the documentation. While you’re at it, explore more posts like this one.

This blog post was created as part of the Mattermost Community Writing Program and is published under the CC BY-NC-SA 4.0 license. To learn more about the Mattermost Community Writing Program, check this out.

Read more about:

Docker documentation OpenAPI QA Testing

Michael Okoko is a software engineer with interests in infrastructure, observability, and low-level systems - including how they work, and the challenges that arise from them.

Read More Engineering Articles