The absolute Minimum Knowledge required to understand the HAL-based API following the Example of MAM Assets CTC

The aim of this document is to give developers information how to approach RESTful APIs based on HAL generally, esp. Avid’s core platform services to access assets, in this case assets of Media Central Asset Management service instances. – Besides general information, parts of this text were the result from solving my own misunderstandings beginning with REST, esp. HAL-based REST APIs.

It should be said, that this document is meant for developers having a solid knowledge of network programming, JSON and HTTP (but not necessarily "web programming" with frameworks like JSP, ASP.NET or PHP). However, this document will also discuss some features of HTTP and the URL-standard, which are less known, but nevertheless exploited to make the HAL-based REST API work.

REST

Generally we have many means to let (local) clients communicate to (remote) services. The story evolved from inter-process communication via shared memory, over socket communication (usually via TCP/IP) with proprietary protocols, over semi-proprietary protocols (CORBA (with the extra protocol IIOP), DCOM, RPC, RMI). After that, standardized communication protocols like XML-RPC and SOAP entered the market. However, the latter technologies have been used successfully in the wild, but they put a heavy burden on administrators and developers:

  • bad compatibility

  • cumbersome, difficult and involved infrastructure

  • to a certain degree long development cycles, the productivity is heavily depending on the tool support

In simple words, to address these issues people went "back to the roots" of communication along the most common protocol stacks, until they found HTTP as the most often used common denominator. – This was effectively one of the findings in Roy Fielding’s doctoral thesis, which is nowadays applied to reinterpret and redesign client-server communication with REST.

It is important to understand, that the means of communication to be found was required to be positioned on the "application layer" of protocol stacks. – This is required, because we need a protocol, which is suited to build an API on top it, remember that API is for "Application Programming Interface"; "lower" protocols have too few semantics to be candidates to act as basis for an API. To drive this point home, HTTP was selected as basis for an API, replacing the other protocols. With HTTP we are standing on the shoulders of giants, because it supports many features out of the box, which we can use directly:

  • HTTP request "verbs" like GET, POST, DELETE etc., reflect (basic) operations with fixed semantics (it basically works like CRUD, but REST is not directly CRUD) ⇒ HTTP is a protocol as well as an API

  • In comparison to, e.g. SOAP or XML-RPC, in REST the URL is rather the method to be called and the method name is not somehow encoded and passed in the payload. I.e. the operation-calling infrastructure is one layer above protocols like SOAP or XML-RPC.

  • Webservers (those are "callees"), which serve HTTP are widespread, cheap or free and could even be implemented relatively easy.

  • Webclients (those are "callers"), which consume HTTP are simple to program on most platforms, because most platforms come with HTTP-support in their core libraries.

    • Basically, firing HTTP-requests as well as analyzing HTTP-traffic can be done without profound programming knowledge using the tools of the OS or free tools.

  • HTTP is a plain text protocol, remember that HTTP is for "Hyper Text Transport Protocol"

    • that makes programming clients even easier, because text-processing is simple

    • errors can be spotted, understood and fixed quickly

HTTP as Plain Text Protocol

In order to exploit HTTP for our purposes, it was required to:

  • Map API-calls to HTTP verbs, i.e. to HTTP requests the verbs are getting repurposed.

  • Define a data protocol.

  • Define how APIs and data work together and how they are linked, so it can be used as hypermedia.

In the following sections these requirements and terms will be explained. And then our approach to meet these requirements is discussed.

General Words on Communication with the Service

Tilman Schlenker held a presentation, or more or less a test run for the toolkit demo, of which you can find the recording here: https://www.cubbyusercontent.com/pl/2016-03-14_17h19m44s+-+Instant+Meeting/_3c116edeabe94928949563d296c31f44#2016-03-14_17h19m44s%20-%20Instant%20Meeting. From that source I took these images showing the layered architecture of the service stack in the Connectivity Toolkit:

Production Setup Development Setup

Callees (Services) and the Upstream Server

According to the image of the service stack, this is in the role of a callee below the bus layer. So, the services are getting registered in the Media Central Bus, therefore they have to use the "Avid Connector API", those services can be inspected via the ACS-Monitor. But the bus is not the means to expose platform services to the clients. The bus lacks some important aspects, which were addressed by another layer on top of the bus, namely the upstream server (upstream) which is delivered with MediaCentral 2.6. A service being provided via the "Avid Connector API", can optionally be "opened" to be reachable via upstream. If it is enabled, upstream will automatically generate and offer a REST interface to this service following a couple of rules, which will be discussed in this paper. As stated above, upstream handles some aspects of communication, which can not be directly handled on the bus layer: security, dispatching, balancing and caching (etc.). Additionally, upstream can be used to allow clients to communicate with multiple independent buses, i.e. such environment w/o having buses in a multizone configuration.

Here you can find upstream-related information, which are esp. of interest for platform service developers:

Callers (Clients)

According to the image of the service stack, the client is in the role of a caller of an API above the NGINX/Dev proxy (e.g. an app running in the browser). Callers will neither directly communicate with the services, nor the bus! Instead they’ll communicate with upstream, which exposes the REST endpoints of the callable services. Put simple, upstream accepts a URL of a REST API and dispatches the call to the correct service via the bus. – Clients, as well 3rd part components/services have to use upstream to communicate with the Avid core services "below" the bus layer. Here you can find information on how upstream maps HTTP requests to bus calls: https://avid-ondemand.atlassian.net/wiki/display/PFENG/Upstream+Service+-+HTTP+API+for+Bus+Services.

Upstream vs Bus

Basically, upstream enables clients to connect to services via the bus. Upstream:

  • Helps Bus services to expose their API via HTTP (In other words, all things HTTP are supposed to go through upstream.)

    • Using HTTP frees clients from using special libraries to access the bus. - They can use any HTTP-library of their choice.

  • Maps synchronous HTTP communication model to asynchronous Bus messaging (using queues etc.)

    • Using a set of rules. ACS Service Wrapper complements Upstream by implementing request / response routing on the service level

    • Handles and authentication

  • Exposes WebSocket API for notifications

Which apps run on the bus?

  • Services communicate with each other via the bus, because they are assumed to be closer to the cluster and need a more or less permanent, immediate and "intimate" connection to each other. So the bus is a "more local" interface for services to communicate with each other.

Which apps run on upstream?

  • Clients, i.e. those entities, which are either apps with a GUI or automation processes, do not need a permanent connection to the bus, also they are more remote to the physical cluster, therefor they’ll connect to upstream to communicate with services.

Entry Point

To enter the API you need the "coordinates" of a service, which is exposed by upstream as well as the credentials:

"Coordinate" Example

API Domain (Name/IP of the upstream’s server i.e. upstream’s host)

"upstream"

Port

8080

Platform Services Root (URI of the service root) and entry point of the API

/apis/avid.mam.assets.access;version=0;realm=BEEF/

Credentials

Either login/password or we already have an access/authorization token in avail.

Note

Special hint for MC|UX programmers: URLs concerning the platform begin with the path-segment "/apis" (i.e. a plural form of api). The singular form "/api" is used in the MC|UX middleware architecture, which is an "application", and not "part of the platform", i.e. "/api" won’t work with the platform.

So let’s assume we have the API domain "upstream" and the port 8080. The platform service, with which we want to work is "/apis/avid.mam.assets.access;version=0", in the realm "BEEF". - For the time being we’ll just replace the placeholder {realm} with the value "BEEF", as it is done above (there is a special URI templating functionality behind that, which we’ll discuss later). - Furthermore we have the access token "MAGIC" (assuming, that we got it from a successful login via IAM). "/apis/avid.mam.assets.access;version=0;realm=BEEF" represents the entry point of the API. To get information about all available operations on that service, we have to call its service root with a simple HTTP GET request:

The raw HTTP Web Request
GET http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/
Authorization: Bearer MAGIC

The GET request shown above is the literal request, which could be issued via Fiddler or similar tools. Generally, issuing HTTP requests, as well as retrieving their responses is very simple; it can be done on virtually any platform (Java, .NET, Cocoa etc.) with integrated means (notice, that HTTP traffic is generally text-based, which makes matters so simple). Mind that the we pass the access token in the header field "Authorization"; well, communication via HTTP is stateless, but we still need to pass some state around.

Note

Advanced information: Bearer-based authentication should not be used, it is only used in the examples of this page. As declared in the text above, this creation of an endpoint is one way to do it. In the wild, the platform will be accessed using HTTPS including a more elaborate authorization-strategy, e.g. based on MC|UX credentials and using cookies. - The authorization will then be a little bit more complex, but still not too complicated.

Potential Clients

Sending an HTTP request is even possible from the terminal/DOS box:

Fire Web Request from the Terminal
#WGET
wget -qO- --header='Authorization: Bearer MAGIC' --header='Accept: application/hal+json' 'http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF'
#CURL (e.g. for macOS)
curl -H 'Authorization: Bearer MAGIC' -H 'Accept: application/hal+json' 'http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF'
Note

In many cases, code executed on the server side will take some time. This can be a problem, if a calling client app awaits a certain response time not being exceeded. "Problem" means, that either the platform of the client app forces a maximum waiting time for the response, or that the responsiveness of a client app’s GUI needs to meet certain ergonomic criteria. The snippets above showing how get requests are issued via WGET and CURL are both working this way, i.e. those calls on the command line will return, when the complete response has arrived. In opposite to this synchronous model, web clients often take the strategy to make asynchronous calls against web services and other kind of I/O. I.e. clients put request code in an event system and get a callback, which is invoked, when the response has arrived. JavaScript it can be done with AJAX (or 3rd party SDKs encapsulating AJAX). Node.js works a little bit different, it makes async calls an idiom (i.e. not only an SDK pattern), calls it "non-blocking I/O", and handles callbacks via an event loop and the consumption of callbacks is abstracted by using a continuation like programming style. - The key difference is, that dispatching of events is done synchronously. (To be frank, it should be said, that Node.js does use threads to put asynchronous I/O into effect. - Results of those threads will then be piped back to the event loop.) In the next example I show, how an async call can be done in Swift/Cocoa using the dataTaskWithRequest() SDK pattern. I passed the request to be issued as well as another piece of code, called "closure" in Swift (it is called "block" in Objective-C, you might also call it lambda), to dataTaskWithRequest(). The closure acts a s callback and will be executed (i.e. "called back"), when the response of the request has arrived:

Asynchronous Web Request with Swift/Cocoa on macOS
#Swift/Cocoa (macOS)
import Cocoa
import XCPlayground

# request represents the call against the web service
let request =
  NSMutableURLRequest(URL:NSURL(string:"http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/")!)
request.addValue("Bearer MAGIC", forHTTPHeaderField: "Authorization")
request.addValue("application/hal+json", forHTTPHeaderField: "Accept")

NSURLSession.sharedSession().dataTaskWithRequest(request) {
    # callback code block begin
    (data, response, error) -> () in
      let result = NSString(data:data!, encoding:NSASCIIStringEncoding)!
      # callback code block end
}.resume()
XCPlaygroundPage.currentPage.needsIndefiniteExecution = true

In this text we’ll not dive into the details of async, or which is the next step, "reactive" programming. Async programming is not trivial. Nevertheless we have provided some examples in Java and Node.js, which show how it works generally. However, async programming seems to be the way to work with HTTP based APIs like CTMS!

Limitations
  • Clients that can only issue simple requests, i.e. those are not able to issue requests with a body and cannot explicitly specify header fields (mind that the Authorization header field is required for bearer-based authentication for our RESTful API to work in the upcoming examples) or cookies (cookies enable alternative authorization-strategies), are not able to communicate with the platform out of the box.

  • Some clients, e.g. Java’s JDK, do not support all HTTP verbs, which are required to communicate via REST. This is the case for the PATCH verb; PATCH requests must be done with a 3rd party HTTP client library, such as Unirest.

  • Calling our RESTful API from JavaScript running in the browser can be tricky. If the JavaScript code in question is not provided from the same source as the API domain, calling upstream will fail due to the mandatory SOP! In order to make this work, either the origins of each need to be the same or JSONP or CORS need to be implemented/configured on the client and server.

For Completeness: a Node.js Client
var http = require("http");

var options = {
    host: "upstream"
    , port: 8080
    , path: "/apis/avid.mam.assets.access;version=0;realm=BEEF/"
    , method: "GET"
    , headers: {
        Authorization: "Bearer MAGIC"
        Accept: "application/hal+json"
    }
};

http.get(
    options
    , function(response) {
        var body = "";
        response.on("data", function(chunk) {
            // reads chunk-wise according to TCP packets
            body += chunk;
        }).on("end", function() {
            console.log(body);
        });
    });

Dissection of the Communication with the Service

HTTP Responses use HAL

Now it’s time to talk about the response of the just sent HTTP GET request ("http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF") to retrieve the service root. What do we get then? In order to enter a REST API we have to request the entry point of that API, and this is what we just have done! The response of the service root describes the API’s entry point, or its "global functions" if you will, in a specific format (or notation). A first look at the responses body/text unleashes it as being of JSON format:

Service Root Resource
{
    "_links": {
        "curies": [
            {
                "href": "http://services.avid.com/apis/assets/{rel}",
                "name": "aa",
                "templated": true
            },
            {
                "href": "http://services.avid.com/apis/locations/{rel}",
                "name": "loc",
                "templated": true
            },
            {
                "href": "http://services.avid.com/apis/search/{rel}",
                "name": "search",
                "templated": true
            },
            {
                "href": "http://services.avid.com/apis/mam/assets/{rel}",
                "name": "ma",
                "templated": true
            }
        ], "self": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF"
        },
        "aa:assets": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets"
        },
        "loc:locations": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/locations"
        },
        "search:searches": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/searches"
        },
        "ma:axf-requests": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/request/axfs"
        }
    },
    "_embedded": {
        "aa:assets": {
            "_links": {
                "self": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets"
                },
                "aa:asset-by-id": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/{id}",
                    "templated": true
                },
                "aa:create-asset": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets",
                    "type": "application/vnd.com.avid.mam.axf+json"
                },
                "aa:update-asset-by-id": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/{id}",
                    "type": "application/vnd.com.avid.mam.axf+json",
                    "templated": true
                },
                "aa:delete-asset-by-id": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/{id}",
                    "templated": true
                }
            }
        },
        "loc:locations": {
            "_links": {
                "self": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/locations"
                },
                "loc:folder-by-id": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/locations/folders/{folderid}",
                    "templated": true
                },
                "loc:root-item": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/locations/folders/1"
                },
                "loc:root-item-templated": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/locations/folders/1{?offset,limit,sort,filter}",
                    "templated": true
                }
            }
        },
        "search:searches": {
            "_links": {
                "search:simple-search": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/searches/quick?search={search}{&offset,limit}",
                    "templated": true
                },
                "search:advanced-search": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/searches/advanced{?offset,limit}",
                    "templated": true
                }
            }
        }
    }
}

To be frank, the responses body looks "involved", the content of the JSON data portion is structured in a complex way. What we see here is data, expressed via another JSON-based protocol, which is put into the HTTP responses body. This protocol is called Hypertext Application Language (HAL). To drive this point home: communication with the upstream is done via HTTP and HAL. HTTP "drives" operations and transports data, HAL represents the data and concretizes the API. So, HAL is used for the communication with the Connectivity Toolkit for the Media Services, in other words it is used for accessing Production, iNews and MAM via the upstream services. The (very dense) HAL specification can be found here: http://stateless.co/hal_specification.html and the more elaborate HAL RFC draft can be found here: https://tools.ietf.org/html/draft-kelly-json-hal-06. The author of HAL, Mike Kelly, says this:

HAL is a little bit like HTML for machines, in that it is generic and designed to drive many different types of application via hyperlinks. The difference is that HTML has features for helping 'human actors' move through a web application to achieve their goals, whereas HAL is intended for helping 'automated actors' move through a web API to achieve their goals.

HAL is one implementation of the HATEOAS (Hypertext As The Engine Of Application State) aspect of the REST paradigm. In this model, operations that are relevant for a given resource are abstracted into link relations, which are a way of identifying URLs that can be used to execute those operations. Link relations form the hypertext network that create a resource space to be navigated. The abstract nature of the link relations isolates server changes from the client code. In HAL, the link relation names are durable, so it’s possible to change the associated URL without necessarily modifying the client. This will be explained further below, and additional information can be found at http://stateless.co/hal_specification.html. In a sense the idea behind HAL is to have a protocol, which defined a standardized way to make JSON a hypertext format, this is done by formalizing links in HAL.

Note

In most cases, when making requests against CTMS' services, we have to make sure to set the "Accept" HTTP header field to the value "application/hal+json" (this header field was set in all the examples used in this text)! This way the client tells the service, that it can deal with "HAL" content in JSON format. - Services can reject requests, w/o the Accept header field value "application/hal+json" with the response "406 - Not Acceptable". There are some exceptions to this rule, where services can respond with other representations.

To see how HAL works in the context of these APIs, we’re going to analyze the previous response.

HAL-formatted objects can have properties, link relations and embedded resources. The service root resource has no data itself, it is just the main entry point to get to other resources, called sub resources from the perspective of the primary resource (service root in this case), via the link relations, therefore it has no properties.

On the first level there is a single field named "_links"; as its name suggests, it just contains links. Within "_links", we have basically two sorts of information: (1) the property "curies", containing another array of links, and (2) a couple of (bare JSON) properties, representing links each. We’ll talk about "curies" in a minute, also notice, that there is another special link "self", which links to the resource of the GET request, we have just requested reflectively, i.e. the URL of the service root! Maybe you doubt the use of this link, but that each resource has a self link is an important feature, when working with embedded resources.

The other links, namely "aa:assets","loc:locations","search:searches" and "ma:axf-requests" link to other resources, which allow access to further groups of functionality

The names of the properties of the other links are following a specific syntax: a prefix (e.g. "aa") followed by a suffix (e.g. "assets") separated with a colon resulting in "aa:assets". - Properties identifying link objects are called "link relations", or short "link rels", i.e. "aa:assets" is a link rel. . Another metaphor for link rels are "method names" or "method selectors", on which different HTTP verbs can be used and on which bodies of different Content-Type (sticking to the metaphor: the method arguments) could be used, we’ll discuss this later. The prefixes of the link rels are directly connected to the names of the CURIEs, listed in the property "curies", CURIE is for "compact URI expression". - What is the idea behind CURIEs?

{
	"_links": {
		"curies": [
			{
				"href": "http://services.avid.com/api/assets/{rel}",
            	"name": "aa",
 				"templated": true
            	}
       	],
    	"aa:assets": {
      		"href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets"
		}
 	}
}

Let’s inspect the CURIE "aa". The CURIE’s name "aa" acts as prefix for all link rels of a common subject if you will. Each CURIE is associated to a URI, specified in the CURIE’s href property, e.g. for "aa" we have "http://services.avid.com/api/assets/{rel}". The CURIE "aa" is also "templated", "templated" means, that we could fill the placeholder "{rel}" in the CURIE’s URI with a link rel’s suffix and get an expanded URL, e.g. for the link rel "aa:assets", we would get the expanded URL "http://services.avid.com/api/assets/assets". - The idea behind those expanded URL is to put an optional documentation of the link rel behind that URL, but in practice, this URL does not have an obligatory meaning, it should however be unique. In practice, and this is also a matter of lengthly discussions concerning HAL, a link rel should never change, i.e. the link rels "aa:assets" and another link rel "bb:assets" are considered different, even if their CURIEs' URIs are equivalent! - That means, that a CURIE does not directly specify a namespace identified by its URI, but it defines a prefix to give link rels a kind of structure. Just keep in mind, that CURIE URIs have no meaning at all, maybe they specify/name the vendor and refer to a documentation, but not more. There is no mechanism like XML-namespacing involved!

Rules for link rels:

  • Link rels should apply CURIEs. Either one of the predefined ones (please see below), or new, maybe vendor-specific ones.

  • Only use hyphens in link rels and CURIE URIs to enhance readability, don’t use underscores, only use lower case letters.

  • For the time being, link rels are evaluated in a case insensitive manner.

Predefined names of the CURIEs:

I have a very simple rule for app developers: Ignore the presence of CURIEs mentally! They have no substantial meaning at all.

For the time being we only discussed the link rels to uniquely designate links. But what do those link rels virtually designate? The JSON-object right from the colon of a link rel property string under the "_links" property is called a link object. (The link rel "curies" does even have an array of link objects associated with it, as explained above, we could list the CURIEs for "aa", "pa" etc. in the "curies" array.) So, we have this general structure of links, link rels and link objects:

Links

Let’s inspect the link object of the link rel "aa:assets":

The Link Object behind the Link Rel "aa:assets"
{
	"href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets"
}

Now let us just follow the link rel’s "aa:assets" href "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets".

Call "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets" via wget
#wget
wget -qO- --header='Authorization: Bearer MAGIC' --header='Accept: application/hal+json' 'http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets'

We’ll get this response:

"http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets"' Response
 {
    "_links": {
        "curies": [
            {
                "href": "http://services.avid.com/apis/assets/{rel}",
                "name": "aa",
                "templated": true
            }
        ], "self": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets"
        },
        "aa:asset-by-id": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/{id}",
            "templated": true
        },
        "aa:create-asset": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets",
            "type": "application/vnd.com.avid.mam.axf+json"
        },
        "aa:update-asset-by-id": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/{id}",
            "type": "application/vnd.com.avid.mam.axf+json",
            "templated": true
        },
        "aa:delete-asset-by-id": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/{id}",
            "templated": true
        }
    }
}

What do we see here:

  • The link rel "self" does exactly refer to the href "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets", we followed to get this response.

  • There is a number of other link rels, which have operation-like names "aa:create-asset" and "aa:delete-asset-by-id".

  • Mind, that some of those link objects are templated, i.e. their property "templated" is set to true and in the values of the href properties we have URI strings, which contain sections marked with placeholders written in curly braces, e.g. "{id}". - A "follower" of these links is meant to fill these placeholders in order to get a "full" URL string, which can then be used in an HTTP request. (Notice, that the placeholders need not to be written as last component of a URI.) We’ll discuss about the meaning of templated URIs and how the placeholders are filled in more detail some sections below, but the basics will get clear as we continue the discussion.

Profiles

URNs? [VTsukur]: URLs are useful if you can guarantee stability of the "location" part of URL. They are more useful because you can point to the profile description with that URL. But if location is not known OR might change OR you do not need resolvable profile URL then URN is more universal. I think at this point it is better to stick to URNs and then publish profile URLs for these URNs when they are available rather than put unstable URLs

Resources

What is in the Resource?

Let’s continue using the HAL response we got from requesting "aa:assets". - The whole HAL-based JSON object is a so called "resource" in HAL lingo. So, the very first request we did to get the service root, responded with the service root resource! Then we started from this resource, following the links to get other resources. The term resource is actually a REST term, and the returned HAL-based JSON object is in reality a "representation of an underlying resource", but HAL uses the term "resource" interchangeably with "representation of the resource".

The aa:assets resource is a collection, meaning that it represents all of the assets in the associated system. The link relations in this resource correspond to operations that apply effectively to the whole asset data store, including creating, retrieving and deleting individual assets.

Using the REST-based API means following the links given in a resource at hand. In the case of the resource "aa:assets" we’ll select the link rel "aa:asset-by-id". We want to follow the href in "aa:asset-by-id"'s link object, therefore we have to:

  • expand the URI template given in the href property to get a fully blown URL by replacing the placeholders by meaningful values

  • fire a HTTP GET request with the expanded URL to get the resource "behind" that URL

All right, in order to expand the URI template in the href, i.e. "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/{id}", we have to know something about the semantics of the link, and what the link, i.e. the HTTP request, will return. In this case, yes I know that from the APIs documentation, I have to replace "{id}" by the id of the asset I want to retrieve. Assuming, that the id is "12374176-1c8d-42b3-9d10-7cf19b0f3fb2", the constructed expanded URL would be "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2". Having that completed URL, we can issue another HTTP GET request, to retrieve the resource of that asset. To have that request work, we have to pass the access token again in the header field "Authorization". E.g. via wget from the command line:

Call aa:asset-by-id via wget
#wget
wget -qO- --header='Authorization: Bearer MAGIC' --header='Accept: application/hal+json' 'http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2'

The HTTP response of that GET request carries another resource in its body: the asset resource. Let’s inspect it!

Response of aa:asset-by-id ⇒ Asset Resource
 {
    "base": {
        "id": "12374176-1c8d-42b3-9d10-7cf19b0f3fb2",
        "type": "asset.VIDEO",
        "systemType": "interplay-mam",
        "systemID": "BEEF"
    },
    "common": {
        "name": "Demo - Updated Asset as of 04-06-16 12:42:58 014_A",
        "creator": "admin",
        "created": "2016-01-28T12:21:18+01:00",
        "modifier": "Service-mamctc",
        "modified": "2016-04-06T12:42:58+02:00",
        "startTC": "00:00:00.000"
    },
    "_links": {
        "curies": [
            {
                "href": "http://services.avid.com/apis/assets/{rel}",
                "name": "aa",
                "templated": true
            },
            {
                "href": "http://services.avid.com/apis/mam/assets/{rel}",
                "name": "ma",
                "templated": true
            }
        ], "self": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2"
        },
        "aa:update-asset": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2;type=asset.VIDEO",
            "type": "application/vnd.com.avid.mam.axf+json"
        },
        "ma:asset-axf": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2;type=asset.VIDEO/axf"
        },
        "aa:attributes": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2;type=asset.VIDEO/attributes"
        },
        "ma:asset-attributes-axf": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2;type=asset.VIDEO/attributes/axf"
        },
        "aa:time-based": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2;type=asset.VIDEO/time-based"
        },
        "ma:asset-strata": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2;type=asset.VIDEO/strata"
        },
        "ma:asset-strata-axf": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2;type=asset.VIDEO/strata/axf"
        },
        "aa:delete-asset": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2"
        },
        "aa:access-by-type-usage-and-protocol": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2;type=asset.VIDEO/access{?filetype,usage,protocol}",
            "templated": true
        },
        "ma:essence-packages": {
            "href": "http:/update:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2;type=asset.VIDEO/essencepackages"
        },
        "profile": {
            "href": "http://meta.avid.com/profiles/mam/asset"
        }
    }
}

Reconsidering the structure of the response of calling "aa:assets", we’ll notice, that this response also is a resource! - Of course it is, because HAL-based APIs only use resources as substantial means of communication. The structure of the "aa:assets" resource and the structure of the "aa:asset" resource only have the "_links" property and esp. the "curies" and "self" properties thereof in common (the hrefs of both "self" link objects do of course differ, because they are different resources). To drive this point home: JSON objects, representing HAL resources can contain links (property "_links") as well as a bunch of arbitrary other properties on the "top level". - In this case the "aa:asset"-resources properties denote more meta information about the individual asset we’ve queried. The properties of a resource are case-sensitive following JSON/JavaScript rules, the casing of the properties might be camelcase or hyphenation (Lisp-case).

We followed the link rel "aa:asset-by-id" to get this resource. In HAL, the original link rel that lead to this resource is implicitly also the type of the resource if you will. In the case of "aa:asset-by-id", this would mean that the resource in the response is an "aa:asset-by-id"-resource, but for matters of clarity, we consider this being an "aa:asset"-resource actually. - As we already discussed, the self link refers to the URL, which was expanded from the original link rels URI template. So the original URL or self link is basically location of that specific resource.

However, we should discuss the link rels in the property "_links" we have in this resource:

  • All link objects, except those in "curies" and "ma:essence-packages" are not templated.

  • The "untemplated" link objects do influence the asset represented by the "aa:asset"-resource itself. I.e. if we follow the link rel "aa:attributes", we’ll get the "aa:attributes" of this ("self") "aa:asset"-resource and if we use "aa:delete-asset", however, this ("self") "aa:asset"-resource will be deleted.

Properties of a Resource

Like every HAL resource we have the property "_links", which we have just discussed, esp. the link rels and link objects. But this time we have more "top-level" properties in that resource (compared to the "aa:assets" resource).

base

The property base refers to another JSON object, that contains a quadruple of common information about the asset, which is represented by this resource. The base property is mandatory for asset objects! I will not go into detail here, the quadruple is just the information, which is needed to have the "Common Object API" working with this data.

common

The additional property common contains other metadata (e.g. "name", "created" (date)).

Other Properties

A resource can also have more specific properties apart from base, common and _links, which will not be discussed here.

Using the API

HATEOAS

What we have done now is basically using the data of one resource as hypermedia to access another resource, i.e. from the "aa:assets" to a specific "aa:asset". The term hypermedia comes into play here, because we have used URIs specified in hrefs, which were provided by the server with a previous (HTTP) response. The way, in which we provided the links was HAL, i.e. the "_links" property, therefor HAL is a hypertext format. This principle of using a REST API is called Hypermedia as the Engine of Application State (HATEOAS). HATEOAS used by a REST client is similar to a human being exploring the web with a web browser following links. The idea is that there is no completely fixed API, but using the correct calls or following the correct links is a matter of discovery via semantics. - It is like a webserver delivering a website to the client’s web browser and the client can select among the links of that website. Another important point is, that we can continue applying HATEOAS with the "aa:asset"-resource, esp. with its links, without prior knowledge of the "initial" URL. → It doesn’t matter from were those links came initially, we can continue working with them. (Well, in or case we have to remember and pass the access token as well, which might break the clean idea of HATEOAS somewhat.)

Note

Advanced information: It should be said, that HATEOAS could also be implemented via HTTP "Link" header fields instead of hypertext as payload. E.g. for forward- and backward navigation there exists RCF 5988 for the link structure.

What we did

So far so good, but another problem is emerging. We understood, that a REST API can be used by "guessing" like a person surfing the web does after the idea of HATEOAS, but how much semantic knowledge is required to use the API? I.e. when we have a link in our hands, what does it really do, which request payload has to be provided and which payload will be present in the response? Well, HTTP provides another means to handle this: verbs (also known as request methods). The HTTP verbs we have used up to now are GET requests. GET requests have the semantics of querying data, or "requesting HTTP content". E.g. this request will fetch the resource specified in the request (here the asset of id "12374176-1c8d-42b3-9d10-7cf19b0f3fb2"):

GET Request
GET http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2 HTTP/1.1

Now lets assume, we want to create a certain resource (e.g. another asset). Following the idea of HATEOAS together with HTTP verbs, we would have to issue an HTTP POST request against the resource in question, incl. an acceptable payload in the body:

POST Request
POST http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/ HTTP/1.1

{...body...}

Will that POST work and do the expected thing, i.e. create a new asset? We decided to use link rels named in an understandable way plus hrefs having the operation in their name combined with HTTP verbs to handle the operations. Argumentation: However, using, or, rather "guessing" only the HTTP verbs to discover functionalities has some serious downsides. We still need to know the allowed HTTP verbs on a resource. To be frank, HTTP provides the OPTIONS verb (to be issued against a specific resource) to give us exactly this information, but we still have no semantic information. I.e. it might be useful to get information about the "createablity" of a resource (then OPTIONS will return the POST verb as one of the allowed verbs), but the knowledge about the bare availability of POST and other verbs doesn’t tell us anything about the payload we have to send. In other words: the supported media type of the content is not known to us. Even worse, the media type of the content of the same resource could vary for different HTTP verbs. - Meaning we could have one format for PUT and another format for PATCH. Therefor the combination of was used.

Instead of only using different HTTP verbs to represent different operations on the same resource, we explicitly define different link rels representing different operations (this approach was taken from the Amazon REST/HAL API). - Then the API-user can directly see, which operations are supported on a resource in its HAL’s "_links" property.

Instead of "guessing", how assets can be created with an HTTP verb, we just name that operation in a clear way in our API. Remember, that we got this response following the link rel "aa:assets":

"aa:assets" Snippet
{
    "_links": {
        "curies": [
            {
                "href": "http://services.avid.com/apis/assets/{rel}",
                "name": "aa",
                "templated": true
            }
        ],
		"self": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets"
        },
        "aa:create-asset": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets",
            "type": "application/vnd.com.avid.mam.axf+json"
        },
       	...
    }
}

With that information we can call the link rel "aa:create-asset", it is fully clear what it does. Fair enough, it is still required, that the caller knows that a POST requests needs to be issued, the combination of the properly named link rel "aa:create-asset" and using it with the POST verb will create the new asset:

Good old POST Request
POST http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/ HTTP/1.1

{...body...}

Another example: deleting assets. In the resource "aa:assets" we’re going to find the link rel "aa:delete-asset-by-id". - Once again, it is completely clear what it does:

"aa:assets" Snippet
{
    "_links": {
        "curies": [
            {
                "href": "http://services.avid.com/apis/assets/{rel}",
                "name": "aa",
                "templated": true
            }
        ],
		"self": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets"
        },
        "aa:delete-asset-by-id": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/{id}",
            "templated": true
        },
		...
    }
}

and finally we use "aa:delete-asset-by-id"'s resolved href like so (once again, the caller needs to know something about the HTTP verb to use, which is DELETE in this case):

DELETE Request
DELETE http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2 HTTP/1.1

So, in the end, the platform API does not make use of "clean" HATEOAS if you will. It does not solely count on the callers' knowledge of the semantics of certain HTTP verbs, instead it provides links having meaningful link rels, which allow proper navigation in the API. Instead of mere HATEOAS, we defined some rules for discovery (link rel names, URI templates etc.), so that callers can make educated guesses, how the API works.

Note

Please notice, that we could also GET the resource "aa:asset" and use the link rel "aa:delete-asset" to delete it!

  • Using "aa:delete-asset" on a specific "aa:asset"-resource is like calling a method "Delete()" on an object of type "Asset".

  • On the other hand using "aa:delete-asset-by-id" on the resource "aa:assets" is rather like calling a global/static function/method "Asset::Delete(id)".

  • Also notice, that "aa:asset" is the singular form of "aa:assets". I.e. the first term addresses an individual object.

So far we have clarified, that we apply convention (link rels named in a meaningful way) in favor to guessing (HTTP verbs). Apart from the semantics of the link rels, we have to clarify the structure of the actual hrefs of the link objects, which also bear semantics, which are also convention driven. Some words on these conventions can be found here: Upstream Service - HTTP API for Bus Services. Generally, links look like this:

http://$apiDomain:$port/apis/$serviceType[;version=$serviceVersion][;realm=$serviceRealm][;region=$regionId][$path][?$queryParameter1=$queryArgument1&$queryParameter2=$queryArgument2]*

We assume $apiDomain having the value "upstream" and $port having the value 8080, hence we’ll omit those values, but implicate their presence.

The URL’s Upstream Layer

/apis/$serviceType[;version=$serviceVersion][;realm=$serviceRealm][;region=$regionId][$path][?$queryParameter1=$queryArgument1&$queryParameter2=$queryArgument2]*

"/apis" is the root to the upstream to access REST-Services. The structure of the first part of every href concerning the platform is defined and standardized on the upstream layer (it is marked in green color in the example above). It contains mandatory parts and optional parts. In short, the part "/apis/$serviceType" is always mandatory, whereby the $serviceType may vary in content. The other parts separated by semicolons are optional and act as parameterization for the specified $serviceType. I.e. those parameters are effective on the specific $serviceType-part of the path, and they are called path-style or matrix parameters. - Theoretically, one could add matrix parameters to any part in the URL’s path, in opposite to query parameters, which are effective on the whole URL!

Table 1. URL Parts

URL Part

What is it?

Examples

/apis

The main, mandatory and fix name of the entry point for all the "calls" against upstream. As already mentioned, URLs concerning the platform begin with the path "/apis" (i.e. a plural form of api). The singular form "/api" is used in the MC|UX middleware architecture, which is not "part of the platform", i.e. "/api" won’t work with the platform/upstream.

FIX

$serviceType

The mandatory specification of the service type.

avid.inews, avid.pam, avid.mam.assets.access

[version=$serviceVersion]

Optional specification of a specific version. If it is not specified, it’ll default to the default version, which was set on the avid.acs.registry

version=0

[realm=$serviceRealm]

Optional specification of a specific realm. If the realm is not specified, it’ll default to "global". (For Interplay related services the realm’s value is the system ID.)

realm=BEFD250969FD4CC3BE03AFBD81D0C797

[region=$regionId]

Optional specification of the zone. If the region is not specified, it’ll default to the local zone "00000000-0000-0000-0000-000000000000".

region=287A058D-9E4B-49A2-8624-179964AF5716

The semantics of the green part of the URL are pretty much clear and defined on the layer of the upstream. Because the API we are presenting here runs on top of upstream, there needs to be more information on the URL, and this is the point, where the $path part of the URL comes into play.

The URL’s Service Layer

/apis/$serviceType[;version=$serviceVersion][;realm=$serviceRealm][;region=$regionId][$path] [?$queryParameter1=$queryArgument1&$queryParameter2=$queryArgument2]*

The $path-part (the blue one) concerns the operations within the specified service. In other words the upstream passes the $-path part to the addressed service. From the perspective of the full URL, the path pointing to a resource is "piggy-packed" on the upstream URL and upstream somehow dispatches the $path to the addresses service.

Table 2. URL Parts
URL Part What is it?

[$path]

This optional part points to a specific resource of the service, i.e. it defines the API of the specific service. If we just set '/' as path, the service root resource will be in the response.

Unfortunately, it is not trivial to define semantics for the $path part. We didn’t discuss the optional $queryParameter-part, but this is part of the semantics for the $path as well. - This lack can, at least partially, be solved by URL-templates, which we’ll discuss later. At least we can enumerate some general rules for the $path-part:

  • Forward slashes are used to express hierarchical relationships, as known from folder hierarchies.

  • (When $path is used in templates, the last character is never a slash.)

  • Two URIs map to two different resources, if they differ in just one character.

  • Only use hyphens in URLs to enhance readability, don’t use underscores, only use lower case letters. URIs are generally case sensitive, but schemes and the authority’s host component are case-insensitive, It should be said, that the usage of lower case letters in URIs is the canonical form for all components.

Table 3. URL Parts
URL Part What is it?

[?$queryParameter1=$queryArgument1&$queryParameter2=$queryArgument2]*

This optional part allows to control the representation of a specific resource, therefore it is specific to the service and resource in question.

Filling in URL Templates

Now its time to dissect the $path-segment of the URL’s service layer. Once again, the $path is the REST request, which will be issued (by the upstream) against the service. In many cases, the $path-segment will be templated in order to pass arguments to a REST call. An alternative to passing arguments to the service via the URL, is using the body of the HTTP request, but this is not an option for a GET request. - This will not be a tutorial discussing the expressiveness of URI-templates and how those will be expanded, the details can found in RFC 6570. Instead some examples of templated URIs and expansions will be presented and explained were appropriate; usually the link rel together with the URI template (esp. its parameter names) is self-explanatory.

Table 4. Example 1: templated path-segment
Link rel Templated URI $path Expansion

aa:asset-by-id

http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/{id}

/assets/{id}

http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2

{id} will only appear as path parameter. It simply represents the id of a resource. The semantic meaning of the id depends on the left hand side of the path. Example: In "/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/{id}", the left hand side of "{id}", i.e. "/assets/" denotes, that the id of a specific asset is to be addressed. The human readable structure of the URL unleashes its semantics: because "…​/assets/" is a plural form of "asset", the complete URL can be read as "the asset resource of the specific asset having the {id} id from the set of all assets".

Table 5. Example 2: templated query parameters
Link rel Templated URI $path Expansion

loc:root-item-templated

http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/locations/folders/1{?offset,limit,sort,filter}

/locations/folders/1{?offset,limit,sort,filter}

http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/locations/folders/1?offset=0&limit=100

Here we have a set of templated query parameters {offset}, {limit}, {sort} and {filter}, which are all optional. We could even call the URI without having the parameters expanded at all.

Limits of URI Templates

As a matter of fact, some of the placeholders could be filled optionally, e.g. {offset}, {limit}, {sort} and {filter} for "loc:root-item-templated". Unfortunately, URI templates provide no way to mark parameters as being optional. Another limitation is, that the expected type of the parameters can not be specified. Callers have to use the correct type by (educated) guessing the correct way to do it, i.e. it makes sense, that {offset} is of type integer.

Some links support the extra query parameter "embed", which is not visible in URI templates. It requests to embed sub resources, which are originally addressable via links, substantially into the primary resource to be present in the response. We’ll discuss embedded resources later. There can also be more optional query parameters, which depend on a certain implementation and have to be documented accordingly. At least some query parameters can already be mentioned:

Query Parameter Value and Semantics Known Usage/Values Example Result

embed

Resources to be embedded. It depends on the semantics of the operation behind the link.

aa:asset-by-id/asset-attributes-axf, it generally awaits a list of resource types to be embedded as local part of their link rel, e.g. pass "asset-attributes-axf" to have a resource of "type" "ma:asset-attributes-axf" embedded.

?embed=asset-attributes-axf

Having embed=asset-attributes-axf specified, a resource object of "type" "aa:asset" is returned, which has a resource object of "type" "ma:asset-attributes-axf" embedded. The embedded resource contains a predefined set of attribute-name/value-pairs (this is documented with that service) which goed beyon the common properties.

attributes

Attributes to be returned in the embedded asset-attributes-axf.

aa:asset-by-id/must be used together with asset-attributes-axf, attributes waits a comma separated list of attribute names

?embed=asset-attributes-axf&attributes=comment

The the extra specified attributes will be put into the embedded "ma:asset-attributes-axf", apart from the predefined ones.

Caching of URLs

From the standpoint of purity, clients should never apply/rely on hard coded ("bookmarked") URLs and they should never build URLs manually, but instead follow link rels every time, e.g. from the root resource to the elementary resource in question. Sure, this puts a greater burden on clients, but this is basically the way hypermedia should work. However, in order to have more efficiency a caching strategy might be appropriate. Questions to be clarified here (Thanks to [Michael Ibbeken].):

When it comes to retrieving the entry point of a service, is that something that I as a consumer of that and possibly other services should be cashing or would I retrieve the entry point every time I need to interact with the service again? If cashing is ok, what is a good granularity of expiring that information?

  • cache per the duration of the current service request (when I am a service myself and currently running in the context of an exposed service method)

  • cache per the life time of the service itself

  • cache per a certain expiration time

  • invalidate the cache when expiration time is due or an exception was detected

Embedded Resources

We just mentioned the concept of embedded resources, which is also a genuine concept of HAL. Therefor some of the links, e.g. "aa:asset-by-id", do also support to specify child resources, which should be embedded in the returned resource. - As already discussed, links, which allow specification of embedded resources are not marked in a special way, instead some known semantics/prior knowledge (hopefully documented) is required.

The idea of embedding is of course interesting to save us from many web requests and getting stuff in bulk instead. Mind that the size of messages are limited on the bus! Let’s Inspect a resource containing an embedded resource by requesting this:

GET http:/upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2?embed=asset-attributes-axf

The returned response looks like this (some parts have been elided for brevity):

{
    "base": { ... },
    "common": { ... },
    "_links": {
        "curies": [
            {
                "href": "http://services.avid.com/apis/mam/assets/{rel}",
                "name": "ma",
                "templated": true
            }
			, ...
        ], "self": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2"
        }
		, ...
    },
    "_embedded": {
        "ma:asset-attributes-axf": {
            "axf": {
                "objects": [
                    {
                        "assetId": "12374176-1c8d-42b3-9d10-7cf19b0f3fb2",
                        "class": "VIDEO",
                        "attributes": [
                            {
                                "name": "MAINTITLE",
                                "value": "Demo - Updated Asset as of 03-18-16 10:04:05 968_A"
                            },
                            {
                                "name": "MODIFICATION_DATETIME",
                                "value": "20160318102954"
                            }
							, ...
                        ]
                    }
                ]
            },
            "_links": {
                "self": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/12374176-1c8d-42b3-9d10-7cf19b0f3fb2;type=asset.VIDEO/attributes/axf"
                }
				, ...
            }
        }
    }
}

Let’s dissect this HAL resource from the JSON’s root. We have the top-level properties "base", "common" and "_links" as always, also mind, that "_links" contains the link rel "curies" among others.

The new property is named "_embedded", as the reader may have noticed, those properties with the "_"-prefix have a predefined meaning in HAL. Within "_embedded" we find the embedded resource as response of the issued request. It is not required to explain the details here, because the structure of the resource is basically equivalent to the stuff we have already discussed, and it also depends on semantics, which have to documented somewhere. However, there is one important point: curies will only be defined in the very top-level resource, e.g. for the embedded resource object of "type" "ma:asset-attributes-axf" the CURIE "ma" is defined on the top-level resource. - It means, that for link rel discovery, it is required to analyze the top-level resource for CURIEs, not the embedded one.

Mind, that embedded resources do also have self links, this is required because we have to be able to associate hrefs to those resources, because we don’t know their origin based on the original request-URL of the "surrounding" resource.

For completeness, it should be mentioned that the JSON-object right from the colon of a link rel property string under the "_embedded" property is called resource object. So, we have this general structure of embedded resources, link rels and resource objects:

Embedded Data

Automatically embedded Resources

Esp. requesting the service root resource, "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/", does always embed the resources "aa:assets", "loc:locations" and "search:searches":

Service Root Resource with embedded Resources
{
    "_links": {
        "curies": [
            {
                "href": "http://services.avid.com/apis/assets/{rel}",
                "name": "aa",
                "templated": true
            },
            {
                "href": "http://services.avid.com/apis/locations/{rel}",
                "name": "loc",
                "templated": true
            },
            {
                "href": "http://services.avid.com/apis/search/{rel}",
                "name": "search",
                "templated": true
            },
			{
                "href": "http://services.avid.com/apis/mam/assets/{rel}",
                "name": "ma",
                "templated": true
            }
        ], "self": {
            "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF"
        },
		...
    },
    "_embedded": {
        "aa:assets": {
            "_links": {
                "self": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets"
                },
                "aa:asset-by-id": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/assets/{id}",
                    "templated": true
                },
                ...
            }
        },
        "loc:locations": {
            "_links": {
                "self": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/locations"
                },
                "loc:folder-by-id": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/locations/folders/{id}",
                    "templated": true
                },
                ...
            }
        },
        "search:searches": {
            "_links": {
                "search:simple-search": {
                    "href": "http://upstream:8080/apis/avid.mam.assets.access;version=0;realm=BEEF/searches/quick?search={search}{&offset,limit}",
                    "templated": true
                },
				...
            }
        }
    }
}

In other words, one could either follow the link rel "aa:assets" from the service root in order to get further links for, e.g. the link rel "aa:asset-by-id", or one could directly follow the link rel "aa:asset-by-id" on the embedded "aa:assets" resource. As can be seen the CURIEs for the embedded resources are all defined on the very top level resource, as shown in the example having "ma:asset-attributes-axf" embedded before.

Optional and Requestable Properties

When dealing with HAL resources, you occasionally stumble over a dilemma: On the one hand, a HAL resource should be complete in the sense that it contains all information that is useful for a client. On the other hand, some pieces of information are probably not supported by all systems or are very expensive to get.

To deal with that, some properties in this documentation are marked as optional. This means:

  • A service can return the property but doesn’t have to.

  • A client must not rely on the existence of the property.

Of course, that puts the burden on the client. It must implement logic to deal with optional properties. To simplify that we declare some properties as optional, but requestable in this documentation. Optional, but requestable properties can be requested with the query parameter insist. This means:

  • A service can return the property but doesn’t have to.

  • If the client requests a property xyz using the query parameter insist=xyz then the service MUST return it.

  • A client that relies on a property xyz MUST request it using insist=xyz when calling a service.

This is typically used for properties that are expensive to get. An example is the property totalNumber when returning a defined range out of a large list of items, e.g. when returning one page of the results of a search, or when returning one page of items in a folder. For a client, the total number is useful to show a UI control for paging. But it can be very expensive to get the total number of items. It can, for example, require additional database queries. So, the performance would suffer if the property is returned all the time, even though a lot of clients won’t need that number at all.

Dealing with Content

!!Notice, that HAL-based JSON can not be converted into XML directly. The problem is that HAL uses JSON-properties having colons in their names like "aa:attributes". - Such a property name can not be used as name for an XML element or attribute, because colon is used as separator for namespace-qualified names in XML!!

Accept Types in Requests

Mind, that the Accept header field value "application/hal+json" must be specified generally!

Content Types in Requests

There are three request verbs accepting a body, which are relevant to us: POST, PATCH and PUT. The verbs PATCH (e.g. link rel "aa:update-asset"), POST (e.g. with link rel "aa:assets") and PUT (e.g. with link rel "aa:asset-by-id") accept content of type "application/vnd.com.avid.mam.axf+json". This vendor-specific content type represents a JSON-variant of interplay | MAM’s AXF XML-structures. Btw. the link rel "aa:asset-by-id" is an example of how the same link rel can be used as a method name to issue a GET (get the asset as HAL response) request as well as a PUT (set an asset’s axf) request.

Content Types and other Header Fields in Responses

Up to now we only discussed HAL as content type of the responses of our requests. And this is also the basic implication: the Content-Type of a response for the platform is HAL by default. This fact can be inspected by reading the header fields of an HTTP response, the default Content-Type header field of a platform response has the value "application/hal+json". All responses of the platform need to have the Content-Type header field specified!

In case of errors, the Content-Type will have the value "application/vnd.avid.problem+json".

The header field "Location" is the substantial data in the response of requesting "aa:create-asset", i.e. the body of the response of "aa:create-asset" is not of interest. Location contains the URL referring to the just created asset, if creation went well.

Content Types in Requests

There are three request verbs accepting a body, which are relevant to us: POST, PATCH and PUT. The verbs PATCH (e.g. link rel "aa:update-asset"), POST (e.g. with link rel "aa:assets") and PUT (e.g. with link rel "aa:asset-by-id") accept content of type "application/vnd.com.avid.mam.axf+json". This vendor-specific content type represents a JSON-variant of Media Central | Asset Management’s AXF XML-structures. Btw. the link rel "aa:asset-by-id" is an example of how the same link rel can be used as a method name to issue a GET (get the asset as HAL response) request as well as a PUT (set an asset’s axf) request.

Restrictions on HTTP Header Fields
  • User defined header fields are not allowed! So, neither the caller nor the callee have to struggle with those.

  • Header field names have to be considered case-insensitive by the standard/RFC. - Upstream converts header field names to lower-case automatically.

Restrictions on HTTP Body Content
  • If POST requests have to be issued, all arguments have to be specified in the body, query parameters should not be used in that case.

  • Upstream supports XML starting with version 5.1.0.

Errors

Up to now, the basic communication protocol we use for the REST-based connectivity toolkit is HTTP and we kind of "re-purposed" HTTP verbs as well as resources to act as means to transport our API calls and data. The same is true for error cases. Our REST-based API also makes use of HTTP error codes to signal error cases to the caller. The Content-Type of an error-response can have the value "application/vnd.avid.problem+json" to signal, that it is an SDK-related error, rather than other "infrastructure" issue. - When the Content-Type is "application/vnd.avid.problem+json", a description/message of the error is awaited in the response body. This text will not go into all the gory details of existing HTTP error codes, and how those should be applied in a clean REST interface. Instead the general idea will be shown. The details about error codes, which may be issued by API calls is documented in the API documentation in detailed manner. In the implementation of the API we applied the proposed error cases as described in https://avid-ondemand.atlassian.net/wiki/download/attachments/25362456/REST%20API%20Guidelines%20.docx?api=v2.

It should be said, that callers of REST APIs should always be aware of the mentioned "infrastructure" issue, which have nothing to do with the requested service per se. Especially timeouts or unreachable endpoints could happen at any time during the application of REST and HATEOAS.

Note

Advanced information: After understanding the most important HTTP idioms, it is time to discuss another way to implement asynchronous communication between client and server. Some sections above we discussed how a client can establish communication with the server in an asynchronous manner in order to have more responsiveness. Sometimes, there is the need to have time consuming operations on the server side, which could lead to timeouts on the client side. (Mind that esp. in case of the platform, synchronized execution on the server side could lead to bus timeouts.) Also initiating asynchronous calls from the client won’t help in this case, because it would only put the waiting-for-result-code into another thread, and then the timeout would just happen in that thread. To handle such situations there also exists an asynchronous "HTTP-communication-pattern" using certain HTTP status codes. It works like so:

  1. When a request was issued, the server could response with a "202 Accepted" status code. This status code means that the request was ok and consumed, but the requested resource was not created yet. In other words, 202 indicates, that the request is still processing. Additionally, the 202 response will carry the header field "Location", which contains the URL to be monitored. This "monitoring URL" refers to a temporary resource and not the actual result!

  2. Then it is the client’s responsibility to poll (i.e. GET) the "monitoring URL" in reasonable intervals: As long as the resulting status code is "200 OK", the asynchronous operation is still running on the server, when the status code does finally have the value "201 Created", the operation finished successfully (of course, the request could also end erroneously responding with an error status code).

    1. If the caller wants to cancel async processing it needs to DELETE the resource behind the "monitoring URL".

  3. The response "201 Created" (some await "303 See Other") will additionally carry another Location header field, which contains the URL to the actual resource representing the result of the original request issued in (1). Additionally its body could contain the created resource as HAL representation.

This pattern provides a solution to deal with time consuming operations on the server to avoid timeouts. The burden on this pattern is clear: it needs to be implemented on the server explicitly.