Choose Your Own Representation

HTTP APIs can be friendlier to clients by providing multiple representations of the same data

By Adam Guest - 28 February 2019

When a client requests data from an API it may only want a subset of available fields, but many APIs only offer a single representation of available data. We can use Content Negotiation for more than just choosing between json or xml (or whatever) and provide exactly what a client asks for.

A Sample API

Many applications use the concept of a collection of items that belong to a user, so here we'll use a collection of favourite colours1.

We can define a simple my-colours endpoint in F# using Giraffe: 

type Colour = {
    Id   : string
    Name : string
    Hex  : string
    RGB  : RGB
    HSL  : HSL
}
and RGB = { Red : int; Green : int; Blue : int }
and HSL = { Hue : int; Saturation : int; Lightness : int }

type MyColoursDto = { Colours : Colour list }

let myColours = [
    { Id   = "abc123"
      Name = "Red"
      Hex  = "#FF0000"
      RGB  = { Red = 255; Green = 0; Blue = 0 }
      HSL  = { Hue = 0; Saturation = 100; Lightness = 50 } }
    { Id   = "def456"
      Name = "Yellow"
      Hex  = "#FFFF00"
      RGB  = { Red = 255; Green = 255; Blue = 0 }
      HSL  = { Hue = 60; Saturation = 100; Lightness = 50 } }
    ]

let webApp =
    choose [
        route "/my-colours" >=> Successful.OK { Colours = myColours } ]

This defines a Colour type that contains a few different properties, and makes a collection of them available via GET request. It returns a hardcoded list of values, but it's just an example so that's ok.

We can also create a simple client to retrieve the data from the endpoint and display it to an end user. The sample swift code is part of an iOS app using URLSession and the Decodable protocol to make things easy: 

struct Colour: Decodable {
    var id: String
    var name: String
    var hex: String
}

var colours: [Colour] = []

private func loadColours() {
    let url = URL(string: "http://localhost:5000/my-colours")

    URLSession.shared.dataTask(with: url!) { [weak self] (data, response, error) in
        guard error == nil else {
            print("Error getting colours")
            return
        }
        guard let content = data else {
            print("Couldn't get any colours")
            return
        }

        do {
            struct ColourDto: Decodable {
                var colours: [Colour]
            }
            let colourData = try JSONDecoder().decode(ColourDto.self, from: content)
            self?.colours = colourData.colours

            DispatchQueue.main.async {
                self?.tableView?.reloadData()
            }
        } catch let err {
            print("Error decoding json", err)
        }
    }.resume()
}

This works pretty well, but we can already see that there's a mismatch between the data that the API provides and the data that the client actually needs - the API returns a bunch of fields that the client doesn't care about. For this sample API it's not much of an issue, but if we had a lot of entries in the list or our colour object was more complex it could become a problem. Fortunately, we have some ways to match our API to the needs of the client.

A New Media Type

We can't just remove values from our API response because there may be other clients relying on that data, but if we provide a way for a client to ask for a different response then we can give each client a response containing the data that it needs.

First, we need to configure Giraffe to support custom media types. We don't need any custom serialization for this, so we can just use the json serializer that's already configured. 

let customJsonContentType mimeType data next (ctx : HttpContext) =
    sprintf "%s; charset=utf-8" mimeType |> ctx.SetContentType
    let serializer = ctx.GetJsonSerializer()
    serializer.SerializeToBytes data
    |> ctx.WriteBytesAsync

type CustomNegotiationConfig (baseConfig : INegotiationConfig) =
    interface INegotiationConfig with
        member __.UnacceptableHandler =
            baseConfig.UnacceptableHandler
        member __.Rules =
            dict [
                "*/*" , json
                "application/json" , json
                "application/vnd.chamook.mini-colours+json",
                customJsonContentType "application/vnd.chamook.mini-colours+json"
                ]

let configureServices (services : IServiceCollection) =
    services.AddGiraffe() |> ignore

    CustomNegotiationConfig(DefaultNegotiationConfig())
    |> services.AddSingleton<INegotiationConfig>
    |> ignore

Then we can create a custom representation of Colours and return it from our endpoint for clients that request the mini version. 

type MiniColour = {
    Id   : string
    Name : string
    Hex  : string
}

type MiniMyColoursDto = { Colours : MiniColour list }

let toMiniColour (c : Colour) = { MiniColour.Id = c.Id; Name = c.Name; Hex = c.Hex }

let getMyColours: HttpHandler =
    fun next ctx ->
        match ctx.TryGetRequestHeader "Accept" with
        | Some x when x.Contains "application/vnd.chamook.mini-colours+json" ->
            Successful.OK
                { MiniMyColoursDto.Colours = myColours |> List.map toMiniColour }
                next
                ctx
        | _ ->
            Successful.OK
                { MyColoursDto.Colours = myColours }
                next
                ctx

With our API configured to return the mini colour response to clients that request it, we can update our app client to send the new Accept header. Handily, for our swift client it's trivial to set an extra header: 

let url = URL(string: "http://localhost:5000/my-colours")!
var request = URLRequest(url: url)
request.setValue("application/vnd.chamook.mini-colours+json", forHTTPHeaderField: "Accept")
        
URLSession.shared.dataTask(with: request) { [weak self] (data, response, error) in
...

Now our client requests a specific response and our API provides it, which is great...for now. However, this does leave our API and client coupled together - if our clients needs change in the future, then our API will also need to be updated even if it could already provide the field, our only options are all or nothing.

Asking For What We Need

We want to find a way that our client can adjust the data it gets from the API as it needs without us needing to make changes to the API. This means that we'll need some way for a client to describe the data that it wishes to receive from the API.

There are a few existing ways of describing the shape of data an API should return to a client, notably GraphQL and JSON:API, so it might be worth considering how to implement this in a way that doesn't necessarily tie us to any particular implementation in the future in case we change preference later on, or especially in the case that we have different API clients with differing needs.

Introspected ReST suggests using MicroTypes as a way of composing a media type. Using that approach, we can add one method of filtering now and then add support for different approaches later without breaking any clients. So we'll need to make the following changes:

Switch our API to use a media type that can support filtering
We'll use a general api media type: application/vnd.chamook.api+json that could work across any apis connected with this one. We don't use something like application/json because we can't guarantee that an API returning plain old json will support filtering (or any other features we wanted to add).
Define a filtering syntax that our client can use
We could use one of the fancy options mentioned earlier. But for now we only want a very simple filter, so a custom field list will probably be easier to implement. Because we're using MicroTypes we could always add support for something fancy like GraphQL later if it turned out we needed it.
Update our client to use the new filtering method
Once this update is done, our client and API should be nicely decoupled for any future changes to filtering.

We could potentially add on ways for the API to describe available filtering options to a client, but if we don't want to go all in on a ReSTful approach with this API we could just leave that as something to be included in documentation - as long as we follow common conventions across our API surface it shouldn't be too problematic.

The Final Form(s)

With that decided, we can update our API with the following code (using the simplest filtering approach, by specifying which fields should be included in the response as a parameter to the media type): 

let tryGetPropertyName<'a> propertyName =
    FSharpType.GetRecordFields typeof<'a>
    |> Array.tryFind
        (fun x -> String.Equals(x.Name, propertyName, StringComparison.OrdinalIgnoreCase))
    |> Option.map (fun x -> x.Name)

let includingRegex = Regex("including=(.*?)(\Z|\0)")

let (|FilteredRepresentation|_|) (accept : string option) =
    match accept with
    | Some a
        when a.StartsWith "application/vnd.chamook.api+json"
            && a.Contains "including=" ->
                includingRegex.Match(a).Groups.[1].Captures.[0].Value.Split ','
                |> Array.map tryGetPropertyName<Colour>
                |> Array.choose id
                |> Array.toList
                |> Some
    | _ -> None

let filterJson (includeFields : string list) (original : JObject) =
    let json = JObject()
    includeFields
    |> List.iter (fun name -> json.[name] <- original.[name])
    json

let getMyColours: HttpHandler =
    fun next ctx ->
        match ctx.TryGetRequestHeader "Accept" with
        | FilteredRepresentation filter ->
            let response = JObject()
            let colourArray =
                myColours
                |> List.map (JObject.FromObject >> filterJson filter)
            response.["colours"] <- JArray(colourArray)
            Successful.OK
                response
                next
                ctx
        | _ ->
            Successful.OK
                { Colours = myColours }
                next
                ctx

Our client needs only minor changes to use the new version: 

struct RGB: Decodable {
    var red: Int
    var green: Int
    var blue: Int
    
    enum CodingKeys: String, CodingKey {
        case red = "Red"
        case green = "Green"
        case blue = "Blue"
    }
}

struct Colour: Decodable {
    var id: String
    var name: String
    var hex: String
    var rgb: RGB
    
    enum CodingKeys: String, CodingKey {
        case id = "Id"
        case name = "Name"
        case hex = "Hex"
        case rgb = "RGB"
    }
}

private func loadColours() {
    let url = URL(string: "http://localhost:5000/my-colours")!
    var request = URLRequest(url: url)
    request.setValue("application/vnd.chamook.api+json; including=id,name,hex,rgb", forHTTPHeaderField: "Accept")
    ...
}

Now our client can specify exactly the fields it wants, and our API will respond appropriately. There are still things that could be improved, but this is enough for now.

The source code for the API and the app client are available on GitHub.