By

Automating your API with JSON Schema

Maintaining API documentation and client libraries can be a pain. Its
time-consuming and everyone seems to forget to update them over time.
Because of this we’ve been looking for new ways of automating our API toolchain
here at Degica.

One of the really cool technologies we have been working with lately is
JSON Schema. Since we’ve had a lot of success
working with the format I wanted to write about how we use it to generate API
documentation, client libraries, and automate tests in our Ruby projects.

What is JSON Schema?

For those unfamiliar with the concept, JSON Schema is a standard for describing
and validating JSON objects, outlined in a series of drafts.
If you’ve ever heard of XML Schema it’s the same concept. JSON Schema is simply a JSON document with rules defining
how to validate JSON Objects. Here’s a simple example:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "name": {
      "description": "Name of the customer",
      "type": "string",
      "pattern": "[a-zA-Z\s]+"
    }
  },
  "required": [
    "name"
  ]
}

Looking at the schema we can see the object must contain a name property, the value type must be a string,
and it must match the regular expression [a-zA-Z\\s]+. Following the rules defined in the schema
we could create an object which would satisfy this schema:

{ "name" : "John Doe" }

One of the great things about JSON Schema is that it is machine-readable.
Once we start creating schema for our API objects it’s very easy to parse and
pull out data when we need it. For example, once we create
a schema we can parse it and use it to generate parts of our API toolchain.
However, we still need a way to describe our API endpoints.

JSON Schema + Hyperschema

As a companion draft to JSON Schema, Hyperschema was created for the purpose of describing
an API by including a links property inside your schema.
Expanding on our previous example we can include the API endpoint inside
the customer schema by adding a links property:

{
  "$schema": "http://json-schema.org/draft-04/hyper-schema",
  "type": "object",
  "properties": {
    "name": {
      "description": "Name of the customer",
      "type": "string",
      "pattern": "[a-zA-Z\\s]+"
    }
  },
  "required": [
    "name"
  ],
  "links": [
    {
      "title": "Create",
      "description": "Create a customer.",
      "href": "/customers",
      "method": "POST",
      "rel": "create",
      "schema": {
        "$ref": "#"
      },
      "targetSchema": {
        "$ref": "#"
      }
    }
  ]
}

If we look at the links property we can see we’ve defined a new route for /customers.
The schema property is the JSON object which the customer endpoint accepts.
The targetSchema defines the JSON object that is returned. Here they both reference
the current schema.

Creating your own Schema

In the real world your JSON Schema will probably be much more complex and difficult
to maintain than the above example. Because of this its a good idea to use tools for scaffolding and validating
JSON Schema.

For our projects we use Prmd which is a Ruby
library maintained by Heroku to quickly generate JSON Schema for API projects.
Using the CLI we can quickly generate a sample schema:

# Create schema for your API objects
$ prmd init customer > schema/customer.json
$ prmd init store > schema/store.json

# Combine them into a single schema
$ prmd combine schema/ > schema.json

# Verify the schema is correct
$ prmd verify schema.json

Before you start creating your own schema I would recommend reading up more on
JSON Schema. Understanding JSON Schema and the offical site
have a lot of great examples.

If you’re looking for more complex examples Heroku maintains their own
JSON Schema.
The JSON Schema for our payment service Komoju is also publically available.

Keeping it in sync

Once you manage to create a JSON schema for your project one can easily forget to update it.
We need a way to keep our schema in sync with our application code.
Being able to test our newly created JSON Schema against code ensures
we don’t forget to update it when we make new changes. In order to do this
will need to assert that the API request and responses match the JSON schema we
created.

Validating API responses

We use the json-schema gem to validate response data returned from our API. Assuming
you’re using RSpec you can create a very simple test matcher to ensure a particular JSON
object matches your JSON Schema:

RSpec::Matchers.define :match_response_schema do |resource|
  match do |response|
    schema = File.read("schema/#{resource}.json")
    data = JSON.parse(response.body)

    JSON::Validator.validate!(schema, data)
  end
end

For example, we can use this matcher to assert our API response matches our customer schema:

it "creates a customer" do
  post "/api/v1/customers", {name: "John Doe"}
  expect(response).to match_response_schema("customer")
end

Validating API requests

Ensuring that our API requests actually match our schema is a little more complicated.
We do this by creating a custom concern for our controllers to validate parameters
against the schema defined in the links property:

module ValidateWithJsonSchema
  extend ActiveSupport::Concern

  included do
    before_action :validate_params
  end

  def resource_name
    controller_name
  end

  def action_name
    params[:action]
  end

  def validate_params
    LinkSchema.new(resource_name, action_name, params).validate!
  end
end

LinkSchema#validate! opens the schema and uses the json-schema gem to valdiate
the link schema for the current action_name. If the parameters don’t match the
links property we defined we raise an exception.

I’ve omitted the code for LinkSchema here as its quite complex in our project. However,
you might want to look at committee
to get an idea of how this would work. Instead of validating it in the controller
you can mount a middleware to validate requests using the committee gem.

Request and Response Examples

All decent developer documentation has request and response examples.
Since we want to generate good documentation will need to record actual
request and response data from our API.

If you’re using RSpec and Rails this is really easy. We can use an
after hook
to execute code after requests specs are executed:

RSpec.configure do |config|
  config.after(:each, :request) do
    file_path = Rails.root.join("schema/examples.json")

    recording = {
      "verb" => request.method,
      "path" => request.path,
      "request_data" => request.request_parameters,
      "response_data" => JSON.parse(response.body),
      "head" => response.status
    }

    key = "#{controller.controller_name}##{controller.action_name}"

    output = {}
    output = JSON.parse(File.read(file_path)) if File.file?(file_path)
    output[key] ||= []
    output[key] << recording

    File.write(file_path, JSON.pretty_generate(output))
  end
end

Since we have access to the request and response instance we can write
the data to a JSON file. The above RSpec hook generates a file schema/examples.json which
looks like the following:

{
  "customers#create": [
    {
      "verb": "POST",
      "path": "/customers",
      "request_data": {
        "name": "John Doe"
      },
      "response_data": {
        "name": "John Doe"
      },
      "head": 200
    }
  ]
}

We can go one step further and tie this in with continuous integration.
Everytime we deploy we can update our examples before generating API documentation
to ensure our examples are up-to-date. We can use these generated examples below.

Generating documentation

Once we have a valid JSON Schema and example data we can render API documentation.
Since JSON is machine-readable its easy to parse the generate documentation
in something like markdown:

<% @schema = JSON.parse(File.read("schema/customer.json")) %>

<%= @schema["title"] %>
---

## <%= @schema["description"] %>

### Attributes

| name | type | description |
| ---- | ---- | ----------- |
<%= @schema["properties"].each do |key, value| %>
| <%= key %> | <%= value["type"] %> | <%= value["description"] %> |
<% end %>

If you’re using Heroku’s interagent JSON Schema format you can use Prmd
to generate API documentation for you.
We use Prmd but with a custom markdown template
to generate the documentation for the Komoju API.

Generating API clients

One of the other great things about JSON Schema is the community of open-source tools built around it.
Right now there are tools for generating ruby, nodejs, and
golang API clients. If you dig around
on github you can probably find generators for other programming languages.

Since JSON Schema is machine-readable you can also build your own client generators
quite easily depending on the programming language.

Conclusion

Automating our API toolchain with JSON Schema has saved us from writing
thousands of lines of code. Auto-generating documentation and API clients
ensures our API toolchain is always up-to-date everytime we deploy. Getting
your project setup with JSON Schema will take time but is worth the investment.

I’ve only briefly touched on some of the ways you can use JSON Schema in your projects.
I encourage readers to read the references below to learn more about the JSON Schema format.

References

  • Elegant API’s with JSON Schema - https://brandur.org/elegant-apis
  • Committee, a collection of ruby middleware for working with JSON Schema - https://github.com/interagent/committee
  • Heroku interagent repository - https://github.com/interagent
  • Understanding JSON Schema - https://spacetelescope.github.io/understanding-json-schema/
  • Official site - http://json-schema.org/
Looking for a Rails developer
Degica is looking for a Rails developer to work with our dev team. Checkout the link for more information.