Automating your API with JSON Schema

Maintaining API documentation and client libraries can be a pain. Its time-consuming and everyone seems to forget to update them over time. Because of this we’ve been looking for new ways of automating our API toolchain here at Degica.

One of the really cool technologies we have been working with lately is JSON Schema. Since we’ve had a lot of success working with the format I wanted to write about how we use it to generate API documentation, client libraries, and automate tests in our Ruby projects.

What is JSON Schema?

For those unfamiliar with the concept, JSON Schema is a standard for describing and validating JSON objects, outlined in a series of drafts. If you’ve ever heard of XML Schema it’s the same concept. JSON Schema is simply a JSON document with rules defining how to validate JSON Objects. Here’s a simple example:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "name": {
      "description": "Name of the customer",
      "type": "string",
      "pattern": "[a-zA-Z\s]+"
    }
  },
  "required": [
    "name"
  ]
}

Looking at the schema we can see the object must contain a name property, the value type must be a string, and it must match the regular expression [a-zA-Z\\s]+. Following the rules defined in the schema we could create an object which would satisfy this schema:

{ "name" : "John Doe" }

One of the great things about JSON Schema is that it is machine-readable. Once we start creating schema for our API objects it’s very easy to parse and pull out data when we need it. For example, once we create a schema we can parse it and use it to generate parts of our API toolchain. However, we still need a way to describe our API endpoints.

JSON Schema + Hyperschema

As a companion draft to JSON Schema, Hyperschema was created for the purpose of describing an API by including a links property inside your schema. Expanding on our previous example we can include the API endpoint inside the customer schema by adding a links property:

{
  "$schema": "http://json-schema.org/draft-04/hyper-schema",
  "type": "object",
  "properties": {
    "name": {
      "description": "Name of the customer",
      "type": "string",
      "pattern": "[a-zA-Z\\s]+"
    }
  },
  "required": [
    "name"
  ],
  "links": [
    {
      "title": "Create",
      "description": "Create a customer.",
      "href": "/customers",
      "method": "POST",
      "rel": "create",
      "schema": {
        "$ref": "#"
      },
      "targetSchema": {
        "$ref": "#"
      }
    }
  ]
}

If we look at the links property we can see we’ve defined a new route for /customers. The schema property is the JSON object which the customer endpoint accepts. The targetSchema defines the JSON object that is returned. Here they both reference the current schema.

Creating your own Schema

In the real world your JSON Schema will probably be much more complex and difficult to maintain than the above example. Because of this its a good idea to use tools for scaffolding and validating JSON Schema.

For our projects we use Prmd which is a Ruby library maintained by Heroku to quickly generate JSON Schema for API projects. Using the CLI we can quickly generate a sample schema:

# Create schema for your API objects
$ prmd init customer > schema/customer.json
$ prmd init store > schema/store.json

# Combine them into a single schema
$ prmd combine schema/ > schema.json

# Verify the schema is correct
$ prmd verify schema.json

Before you start creating your own schema I would recommend reading up more on JSON Schema. Understanding JSON Schema and the offical site have a lot of great examples.

If you’re looking for more complex examples Heroku maintains their own JSON Schema. The JSON Schema for our payment service Komoju is also publically available.

Keeping it in sync

Once you manage to create a JSON schema for your project one can easily forget to update it. We need a way to keep our schema in sync with our application code. Being able to test our newly created JSON Schema against code ensures we don’t forget to update it when we make new changes. In order to do this will need to assert that the API request and responses match the JSON schema we created.

Validating API responses

We use the json-schema gem to validate response data returned from our API. Assuming you’re using RSpec you can create a very simple test matcher to ensure a particular JSON object matches your JSON Schema:

RSpec::Matchers.define :match_response_schema do |resource|
  match do |response|
    schema = File.read("schema/#{resource}.json")
    data = JSON.parse(response.body)

    JSON::Validator.validate!(schema, data)
  end
end

For example, we can use this matcher to assert our API response matches our customer schema:

it "creates a customer" do
  post "/api/v1/customers", {name: "John Doe"}
  expect(response).to match_response_schema("customer")
end

Validating API requests

Ensuring that our API requests actually match our schema is a little more complicated. We do this by creating a custom concern for our controllers to validate parameters against the schema defined in the links property:

module ValidateWithJsonSchema
  extend ActiveSupport::Concern

  included do
    before_action :validate_params
  end

  def resource_name
    controller_name
  end

  def action_name
    params[:action]
  end

  def validate_params
    LinkSchema.new(resource_name, action_name, params).validate!
  end
end

LinkSchema#validate! opens the schema and uses the json-schema gem to valdiate the link schema for the current action_name. If the parameters don’t match the links property we defined we raise an exception.

I’ve omitted the code for LinkSchema here as its quite complex in our project. However, you might want to look at committee to get an idea of how this would work. Instead of validating it in the controller you can mount a middleware to validate requests using the committee gem.

Request and Response Examples

All decent developer documentation has request and response examples. Since we want to generate good documentation will need to record actual request and response data from our API.

If you’re using RSpec and Rails this is really easy. We can use an after hook to execute code after requests specs are executed:

RSpec.configure do |config|
  config.after(:each, :request) do
    file_path = Rails.root.join("schema/examples.json")

    recording = {
      "verb" => request.method,
      "path" => request.path,
      "request_data" => request.request_parameters,
      "response_data" => JSON.parse(response.body),
      "head" => response.status
    }

    key = "#{controller.controller_name}##{controller.action_name}"

    output = {}
    output = JSON.parse(File.read(file_path)) if File.file?(file_path)
    output[key] ||= []
    output[key] << recording

    File.write(file_path, JSON.pretty_generate(output))
  end
end

Since we have access to the request and response instance we can write the data to a JSON file. The above RSpec hook generates a file schema/examples.json which looks like the following:

{
  "customers#create": [
    {
      "verb": "POST",
      "path": "/customers",
      "request_data": {
        "name": "John Doe"
      },
      "response_data": {
        "name": "John Doe"
      },
      "head": 200
    }
  ]
}

We can go one step further and tie this in with continuous integration. Everytime we deploy we can update our examples before generating API documentation to ensure our examples are up-to-date. We can use these generated examples below.

Generating documentation

Once we have a valid JSON Schema and example data we can render API documentation. Since JSON is machine-readable its easy to parse the generate documentation in something like markdown:

<% @schema = JSON.parse(File.read("schema/customer.json")) %>

<%= @schema["title"] %>
---

## <%= @schema["description"] %>

### Attributes

| name | type | description |
| ---- | ---- | ----------- |
<%= @schema["properties"].each do |key, value| %>
| <%= key %> | <%= value["type"] %> | <%= value["description"] %> |
<% end %>

If you’re using Heroku’s interagent JSON Schema format you can use Prmd to generate API documentation for you. We use Prmd but with a custom markdown template to generate the documentation for the Komoju API.

Generating API clients

One of the other great things about JSON Schema is the community of open-source tools built around it. Right now there are tools for generating ruby, nodejs, and golang API clients. If you dig around on github you can probably find generators for other programming languages.

Since JSON Schema is machine-readable you can also build your own client generators quite easily depending on the programming language.

Conclusion

Automating our API toolchain with JSON Schema has saved us from writing thousands of lines of code. Auto-generating documentation and API clients ensures our API toolchain is always up-to-date everytime we deploy. Getting your project setup with JSON Schema will take time but is worth the investment.

I’ve only briefly touched on some of the ways you can use JSON Schema in your projects. I encourage readers to read the references below to learn more about the JSON Schema format.

References

• Elegant API’s with JSON Schema - https://brandur.org/elegant-apis
• Committee, a collection of ruby middleware for working with JSON Schema - https://github.com/interagent/committee
• Heroku interagent repository - https://github.com/interagent
• Understanding JSON Schema - https://spacetelescope.github.io/understanding-json-schema/
• Official site - http://json-schema.org/