Upstream Service - HTTP API for Bus Services

TOC

Goals

  • Provide mapping of:
    • Incoming HTTP requests into outgoing bus messages targeted to the requested bus service

Overview

  • Upstream routes the message to the correct service with a regular bus op according to the rules in serviceInfo

Example

  1. Upstream creates a message for avid.iam with:
    1. full HTTP info in context.http.request
    2. serviceType=avid.iam
    3. Realm and zone are defaulted, unless provided explicitly with realm and region matrix parameters.
    4. op=createToken
    5. Headers of the HTTP request goes to context.http.request
    6. The IAM token which is associated with the request goes to context.identity
  2. Service using ProxyBAL dispatches the op to the right call:
    1. Service returns resultSet, errorSet and potentially context.http.response.
  3. Upstream responds to the client based on whatever is described in the response:
    1. Including the body, status code and header changes from context.http.response (if available).

HTTP API

URL Template

/apis/$serviceType[;realm=$serviceRealm][;version=$serviceVersion][;region=$regionId][/$path][?$queryParameter%i=queryParameterValue%i]*

Methods

HTTP methods mentioned in the following specifications MUST be supported unless noted otherwise (see below):

  • Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content - RFC 7231 (obsoletes RFC 2616)
  • PATCH Method for HTTP - RFC 5789

Complete method list supported by the Upstream:

  • GET
  • POST
  • PUT
  • DELETE
  • PATCH

TRACE method MUST be disabled for security reasons. CONNECT method is not used in APIs, so there is no immediate need to support it.

Request Mapping

Service Type

Value of the service type MUST be determined as the value of the respective $serviceType path segment in the request URI. This path segment is mandatory. If value of the $serviceType path segment is not specified or does not correspond to the existing service on the bus then Upstream MUST respond with 404 Not Found.

Examples
HTTP Request target Bus service type
/apis N/A → 404 Not Found
/apis/3rd.party/endpoint?query=value 3rd.party
/apis/avid.asset.storage avid.asset.storage
/apis/avid.iam/principals avid.iam
/apis/non-existing-service-type N/A → 404 Not Found
Service Realm

Value of the service realm MUST be determined as the value of the respective $serviceRealm matrix parameter in the request URI. If $serviceRealm matrix parameter is not specified then default service realm MUST be used. Default value of the service realm is global. If value of the $serviceRealm matrix parameter is not a valid value then Upstream MUST respond with 504 Gateway Time-out.

Examples
HTTP Request target Bus service realm
/apis/avid.assets global
/apis/avid.delivery;realm=bac2ea20-2f76-11e4-8c21-0800200c9a66/principals bac2ea20-2f76-11e4-8c21-0800200c9a66
Service Version

Value of the service version MUST be determined as the value of the respective $serviceVersion matrix parameter in the request URI. If value of the $serviceVersion matrix parameter is not a valid number then Upstream MUST respond with 504 Gateway Time-out.

Examples
HTTP Request target Bus service version
/apis/avid.asset.storage default version set for the service in the registry
/apis/avid.iam;version=2/principals 2
/apis/3rd.party;version=abc/endpoint?query=value N/A → 504 Gateway Time-out
Region

Bus message MUST be sent to a zone with the identifier determined as the value of the respective $regionId matrix parameter in the request URI. If $regionId matrix parameter is not specified then message MUST be sent to the local zone. If value of the $regionId matrix parameter is not a valid value then Upstream MUST respond with 504 Gateway Time-out.

Examples
HTTP Request target Bus Zone
/apis/avid.assets local zone if service exists in the local zone, otherwise any of the remote zones
/apis/avid.delivery;region=f9823030-2f77-11e4-8c21-0800200c9a66/principals f9823030-2f77-11e4-8c21-0800200c9a66
/apis/non-existing-service-type N/A → 504 Gateway Time-out
Operation

Upstream will look for the operation in the target service with REST description which corresponds to the meta information of the HTTP request. The name of the operation will be used as a value for op property in the bus message.

Method

Method name MUST be passed in context.http.request.method property of the outgoing bus message as a simple string value.

Examples
HTTP Method Bus Message
GET
{
  [...],
  "context": {
    "http": {
        "request": {
            [...],
            "method": "GET"
         }
    }
    [...]
  },
  [...]
}
POST
{
  [...],
  "context": {
    "http": {
        "request": {
            [...],
            "method": "POST"
         }
    }
    [...]
  },
  [...]
}
Query Parameters

All query parameters passed in the HTTP request target MUST be mapped onto the paramSet object as key - value pairs where key is a query parameter name and value is a string value of the parameter. If query parameter is specified several times then value MUST be passed as an array of strings. Query parameters with prefix name "_avid" IS NOT allowed and wouldn't be passed to bus.

Examples
Query Parameters Bus Message
N/A
No queryParams property:
{
  [...],
  "paramSet": {
    ...
  },
  [...]
}
?offset=0&limit=25
{
  [...],
  "paramSet": {
    [...]
      "offset": "0",
      "limit": "25"
    [...]
  },
  [...]
}
?key=value1&key=value2
{
  [...],
  "paramSet": {
    [...],
      "key": [ "value1", "value2" ]
    [...]
  },
  [...]
}
?key
{
  [...],
  "paramSet": {
    [...],
      "key": ""
    [...]
  },
  [...]
}
?key=
{
  [...],
  "paramSet": {
    [...],
      "key": ""
    [...]
  },
  [...]
}
HTTP Message Metadata

Metadata of HTTP request MUST be captured and stored in the bus message context in the form of JSON object residing under context/http/request path:

IMPORTANT:

Upstream version 4.7+ performs normalization of all incoming HTTP header names (lowercases them) before putting them into bus message. According to RFC7230 header field names are case-insensitive. Making headers lowercase makes service code easier to write and less error-prone since service does not need to normalize the case - platform is doing that already.

Metadata Context Path in Bus Message Data Type Example
Request
method
http.request.method String
GET /apis/avid.iam/principals?offset=1 HTTP/1.1
[...]
{
  [...],
  "context": {
    "http": {
      "request": {
        [...],
        "method": "GET",
        [...]
      }
    }
  },
  [...]
}
Request
target
http.request.target String
GET /apis/avid.iam/principals?offset=1 HTTP/1.1
[...]
{
  [...],
  "context": {
    "http": {
      "request": {
        [...],
        "target": "/apis/avid.iam/principals?offset=1",
        [...]
      }
    }
  },
  [...]
}
Request
headers
http.request.headers Object

GET /apis/avid.iam/principals HTTP/1.1 Host: api.everywhere.avid.com Connection: keep-alive Accept: application/json; q=0.9, application/hal+json;q=1 User-Agent: Chrome/36.0.1985.125 Safari/537.36

{
  [...],
  "context": {
    "http": {
      "request": {
        [...],
        "headers": {
          "Host": "api.everywhere.avid.com",
          "Connection": "keep-alive",
          "Accept": "application/json; q=0.9, application/hal+json;q=1",
          "User-Agent": "Chrome/36.0.1985.125 Safari/537.36"
        },
        [...]
      }
    }
  },
  [...]
}
**Note that upstream version > 4.7.0 will lowercase all header names** Example:
{
  [...],
  "context": {
    "http": {
      "request": {
        [...],
        "headers": {
          "host": "api.everywhere.avid.com",
          "connection": "keep-alive",
          "accept": "application/json; q=0.9, application/hal+json;q=1",
          "user-agent": "Chrome/36.0.1985.125 Safari/537.36"
        },
        [...]
      }
    }
  },
  [...]
}
Client
address
http.request.clientAddress String
{
  [...],
  "context": {
    "http": {
      "request": {
        [...],
        "clientAddress": "172.22.34.89"
      }
    }
  },
  [...]
}
Base URL
template
http.request.baseUrlTemplate String
{
  [...],
  "context": {
    "http": {
      "request": {
        [...],
        "baseUrlTemplate": "https://api.us-east-1.everywhere.avid.com/apis{/serviceType}{;version,realm,region}{+path}"
      }
    }
  },
  [...]
}
Compound Example
HTTP Bus
Request
GET /apis/avid.iam/principals HTTP/1.1
Host: api.everywhere.avid.com
Connection: keep-alive
Accept: application/json; q=0.9, application/hal+json;q=1
User-Agent: Chrome/36.0.1985.125 Safari/537.36
{
  "serviceType": "avid.iam",
  "serviceRealm": "global",
  "serviceVersion": 0,
  "op": [...],
  "context": {
    "identity": [...],
    "http": {
      "request": {
        "version": "1.1",
        "method": "GET",
        "target": "/apis/avid.iam/principals",
        "headers": {
          "Host": "api.everywhere.avid.com",
          "Connection": "keep-alive",
          "Accept": "application/json; q=0.9, application/hal+json;q=1",
          "User-Agent": "Chrome/36.0.1985.125 Safari/537.36"
        },
        "baseUrlTemplate": "https://api.everywhere.avid.com/apis{/serviceType}{;version,realm,region}{+path}",
        "clientAddress": "172.22.34.89"
      }
    }
  },
  "paramSet": [...]
}
Body

HTTP request body MUST be mapped onto the paramSet object under body key as a JSON object holding content and encoding.

IMPORTANT: Upstream as a separate service does not support Transfer-encoding: chunked. Most of the developers work with Upstream via Nginx proxy service and for them Transfer-encoding: chunked will transfer body, but instead sending data in a series of chunks it would be sent as a whole.

IMPORTANT: HTTP request MUST supply Content-Length header so that the body is read by the Upstream. Otherwise request body will not be available.

IMPORTANT: Upstream allows to pass XML payloads in requests and responses. To pass XML payload request must have one of the following content types application/xml, application/+xml*, text/xml, text/+xml*.

Encoding

Body encoding is stored under encoding key as a string in the body object. Encoding defines how content is represented in the bus message:

  • json - content is passed as plain vanilla JSON. Value is passed as an inline JSON object in the bus message. This is a default encoding for JSON payloads (application/json, application/*+json and application/json-patch).

NOTE: Encoding of the body data in the bus message is NOT the same as Content-Encoding header field in HTTP request.

Examples
HTTP Request Bus Messages
POST /apis/avid.iam/principals HTTP/1.1
Content-Type: application/json; charset=utf-8
Content-Length: 78
[...]
{
  "entity": {
    "kind": "user",
    "alias": "user1@google.com"
  }
}
{
  [...],
  "paramSet": {
    [...],
    "body": {
      "encoding": "json",
      "entity": {
          "kind": "user",
          "alias": "user1@google.com"
      }
    }
    [...],
  },
  [...]
}
POST /apis/hal.acceptor/giveMeContent HTTP/1.1
Content-Type: application/hal+json; charset=utf-8
Content-Length: 71
[...]
{
  "\_links": {
    "next": {
      "href": "/page=2"
    }
  }
}
{
  [...],
  "paramSet": {
    [...],
    "body": {
      "encoding": "json",
      "data": {
        "\_links": {
          "next": {
            "href": "/page=2"
          }
        }
      }
    }
    [...],
  },
  [...]
}
POST /apis/upload.service/text HTTP/1.1
Content-Type: application/octet-stream
Content-Length: 5
[...]
abcde
{
  [...],
  "paramSet": {
    [...],
    "body": {
      "encoding": "base64",
      "data": "YWJjZGU="
    }
    [...],
  },
  [...]
}
POST /apis/service/empty HTTP/1.1
Content-Type: application/json; charset=utf-8
[...]
{
  [...],
  "paramSet": {
    [...]
  },
  [...]
}
GET /apis/avid.iam/principals HTTP/1.1
[...]
{
  [...],
  "paramSet": {
    [...]
  },
  [...]
}
POST /apis/avid.iam/principals HTTP/1.1
Content-Type: application/json; charset=utf-8
Content-Length: 2
[...]
[]
400 Bad Request
(body content passed in is not a valid JSON)

Response Mapping

HTTP Message Metadata

Metadata of HTTP response MUST be captured from the bus message context in the form of JSON object residing under http/response path overriding any default metadata values and assumptions made by the upstream:

Metadata Context Path in Bus Message Data Type Example Comments
Response status code http.response.status String
or
Number
{
  [...],
  "context": {
    "http": {
      "response": {
        "status": 204,
        [...]
      }
    }
  },
  [...]
}
HTTP/1.1 204 No Content
[...]
Optional field. Default values:
  • 500 if there is an errorSet
  • 200 if there is a response body
  • 204 if there is no response body
Response headers http.response.headers Object
{
  [...],
  "context": {
    "http": {
      "response": {
        [...],
        "headers": {
          "Content-Type": "application/hal+json",
          "Content-Encoding": "gzip"
        }
      }
    }
  },
  [...]
}
HTTP/1.1 [...]
[...]
Content-Type: application/hal+json
Content-Encoding: gzip
Body

HTTP response body MUST be mapped from the resultSet or errorSet of the bus message:

  • If errorSet is present then it MUST be used as a source of response body. Presence of errorSet means there was (at least one) problem when processing the request.
  • Else resultSet MUST be used.
Errors

In case errorSet is present, its first error MUST be mapped to the response body with the rules outlined below:

{
  "serviceType": "avid.service",
  "serviceRealm": "global",
  "serviceVersion": 1,
  "op": "someOperation",
  "context": ...
  "errorSet": [
    {
      "code": "internal/avid.service/global/1/I0001",
      "details": "Matching method for http request [missing/resource] not found",
      "params": {
        "resource": "missing/resource"
      },
      "message": "Requested resource missing/resource not found",
      "incident": "625c09c7-0a3f-4ffc-b834-bfc773236622",
      "severity": "ERROR",
      "status": 404
    },
    ...
  ]
}
Bus Message HTTP Response
HTTP/1.1 404 Not Found
Content-Type: application/vnd.avid.error+json
...

{ "status": 404, "code": "internal/avid.service/global/1/I0001", "params": { "resource": "missing/resource" }, "message": "Requested resource missing/resource not found", "incident": "625c09c7-0a3f-4ffc-b834-bfc773236622", "exchange": "1a6a5fbf-9109-4c08-87cb-2d2b89ab89bc" }

  • application/vnd.avid.problem+json is used as a media type for representing errors.
  • First error in the errorSet "becomes" the representation of error resource.
    • The motivation is to have the simplest and shortest form for the major use case.
    • Otherwise client would need to explicitly dive into $.errorSet[0] and think in terms of multiple errors each and every time instead of having instant error access.
  • status is always added to the response even if bus message does not carry one.
    • API of ACS BAL prior to version 3.2 does not allow to construct error with status. ACS BAL 3.2 and above deprecated this API, but still supports it. If status is NOT present in bus error:
      • Upstream 4+ sets status to 500 unless it is overridden by context.http.response.status
      • Since code was used in some cases to communicate status code in pre ACS BAL 3.2 API, Upstream 4+ will set status to 404 if code is set to "404" and no context.http.response.status is specified.
  • exchange is added to the representation to trace back unique HTTP request / response pair.
  • details are stripped (still logged to the Upstream main log).
    • The motivation behind default behavior is to avoid exposure of sensitive data.
  • severity is stripped (still logged to the Upstream main log).
    • The motivation behind default behavior is to avoid exposure of data that is of no help to the client.

Only the first error is considered. Others are ignored.

Bus Message HTTP Response Entity
{
  "serviceType": "avid.service",
  "serviceRealm": "global",
  "serviceVersion": 1,
  "op": "operation",
  "context": ...
  "errorSet": [
    {
      "code": "internal/avid.service/global/1/I0001",
      ...
    },
    {
      "code": "internal/avid.service/global/1/I0002",
      ...
    },
    ...
  ]
}
{
  "code": "internal/avid.service/global/1/I0001",
  ...
}

Mapping of resultSet is based on the following rules:Results

  • If there is a key under resultSet/body/data path then respective value MUST be used as a source of response body.
  • Else no response body will be provided.
Bus Message HTTP Response Body
{
  [...],
  "resultSet": {
    "body": {
      "data": {
        "key": "value"
      }
    }
  }
}
{
  "key": "value"
}
{
  [...],
  "resultSet": {
    "key": "value"
  }
}
(no body)

Body encoding MAY be stored under resultSet/body/encoding key as a string. If encoding is not specified then:

  • json encoding if resultSet/body/data is a JSON object.

NOTE: Encoding of the body data in the bus message is NOT the same as Content-Encoding header field in HTTP response. Source value gets decoded according to the encoding rules. Upstream SHOULD NOT try to validate content negotiation rules - responsibility SHOULD be a responsibility of the service.

Bus Message HTTP Response Body
{
  [...],
  "resultSet": {
    "body": {
      "data": {
        "key": "value"
      }
    }
  }
}
{
  "key": "value"
}
{
  [...],
  "resultSet": {
    "body": {
      "data": "string value"
    }
  }
}
string value
{
  [...],
  "resultSet": {
    "body": {
      "data": 5
    }
  }
}


5
{
  [...],
  "resultSet": {
    "body": {
      "data": [1, 2, 3, 4, 5]
    }
  }
}
[1, 2, 3, 4, 5]
{
  [...],
  "resultSet": {
    "body": {
      "data": true
    }
  }
}
true
{
  [...],
  "resultSet": {
    "body": {
      "data": null
    }
  }
}
(no body)
{
  [...],
  "resultSet": {
    "body": {
      "data": {}
    }
  }
}

{
  [...],
  "resultSet": {
    "body": {
      "encoding": "json",
      "data": {}
    }
  }
}

{
  [...],
  "resultSet": {
    "body": {
      "encoding": "base64",
      "data": "c3RyaW5nIHZhbHVl"
    }
  }
}
string value
(base64 decoded from data)
{
  [...],
  "resultSet": {
    "body": {
      "encoding": "json"
    }
  }
}
(no body)
{
  [...],
  "resultSet": {
    "body": {
      "encoding": "string"
    }
  }
}
(no body)
{
  [...],
  "resultSet": {
    "body": {
      "encoding": "base64"
    }
  }
}
(no body)
{
  [...],
  "resultSet": {
    "body": {
      "encoding": "json",
      "data": "string value"
    }
  }
500 Internal Server Error
(json encoding assumes that data value is a JSON object)
{
  [...],
  "resultSet": {
    "body": {
      "encoding": "string",
      "data": {
        "key": "value"
      }
    }
  }
}
{"key":"value"}
(*string* encoding does automatic conversion of *data* value to a string)
{
  [...],
  "resultSet": {
    "body": {
      "encoding": "base64",
      "data": {
        "key": "value"
      }
    }
  }
}
*500 Internal Server Error* (*base64* encoding assumes that *data* value is a string)
Examples

Response Metadata Mapping

HTTP Bus

Response
HTTP/1.1 200 OK
Location: httрs://api.everywhere.avid.com/apis/avid.iam/principals/123
Content-Type: application/hal+json

{ "entity": { "kind": "user", "alias": "user1@google.com", "created": "20140809T183142-03", [...] } }

{ "serviceType": "avid.iam", "serviceRealm": "global", "serviceVersion": 0, "op": "findPrincipalById", "context": { "http": { "response": { "status": 201, "headers": { "Location": "httрs://api.everywhere.avid.com/apis/avid.iam/principals/123", "Content-Type": "application/hal+json" } } } }, "resultSet": { "body": { "encoding": "json", "data": { "entity": { "kind": "user", "alias": "user1@google.com", "created": "20140809T183142-03", [...] } } } } }

Retrieving Principals

HTTP Bus
Request
GET /apis/avid.iam/principals?offset=0&limit=25\
&orderBy=name&order=asc&entity=%5B%7Bfield%3A+%22kind%22%2Cvalue%3A+%22user%22%7D%5D HTTP/1.1
[...]
{
  "serviceType": "avid.iam",
  "serviceRealm": "global",
  "serviceVersion": 0,
  "op": "findPrincipalById",
  "context": {
    "identity": [...],
    "http": {
      "request": {
        "version": "1.1",
        "method": "GET",
        "target": "/apis/avid.iam/principals?offset=0&limit=25\
  &orderBy=name&order=asc&entity=%5B%7Bfield%3A+%22kind%22%2Cvalue%3A+%22user%22%7D%5D",
        [...]
      }
    }
  },;
  "paramSet": {
      "offset": "0",
      "limit": "25",
      "orderBy": "name",
      "order": "asc",
      "entity": "[{field: \"kind\",value: \"user\"}]"
  }
}

Response
HTTP/1.1 200 OK
[...]
{
  "entity": [
    {
      "field1": "value1",
      "field2": "value2"
    },
    {
      "field1": "value3",
      "field2": "value4"
    }
  ]
}
{
  "serviceType": "avid.iam",
  "serviceRealm": "global",
  "serviceVersion": 0,
  "op": "operation",
  "resultSet": {
    "body": {
      "encoding": "json",
      "data": {
        "entity": [
          {
            "field1": "value1",
            "field2": "value2"
          },
          {
            "field1": "value3",
            "field2": "value4"
          }
        ]
      }
    }
  }
}

Creating Principal(s)

HTTP Bus
Request
POST /apis/avid.iam/principals HTTP/1.1
Content-Type: application/json; charset=utf-8
Content-Length: 78
[...]

{ "entity": { "kind": "user", "alias": "user1@google.com", [...] } }

{
  "serviceType": "avid.iam",
  "serviceRealm": "global",
  "serviceVersion": 0,
  "op": "operation",
  "context": {
    "identity": [...],
    "http": {
      "request": {
        "version": "1.1",
        "method": "POST",
        "target": "/apis/avid.iam/principals",
        [...]
      }
    }
  },
  "paramSet": {
    "body": {
      "encoding": "json",
      "data": {
        "entity": {
          "kind": "user",
          "alias": "user1@google.com",
          [...]
        }
      }
    }
  }
}

Response
HTTP/1.1 201 Created
Location: httрs://api.everywhere.avid.com/apis/avid.iam/principals/123
[...]

{ "entity": { "kind": "user", "alias": "user1@google.com", "created": "20140809T183142-03", [...] } }

{
  "serviceType": "avid.iam",
  "serviceRealm": "global",
  "serviceVersion": 0,
  "op": "operation",
  "context": {
    "http": {
      "response": {
        "status": 201,
        "headers": {
          "Location": "httрs://api.everywhere.avid.com/apis/avid.iam/principals/123"
        }
      }
    }
  },
  "resultSet": {
    "body": {
      "encoding": "json",
      "data": {
        "entity": {
          "kind": "user",
          "alias": "user1@google.com",
          "created": "20140809T183142-03",
          [...]
        }
      }
    }
  }
}

Retrieving Principal

HTTP Bus
Request
GET /apis/avid.iam/principals/123?regionId=123 HTTP/1.1
[...]
{
  "serviceType": "avid.iam",
  "serviceRealm": "global",
  "serviceVersion": 0,
  "op": "operation",
  "context": {
    "identity": [...],
    "http": {
      "request": {
        "version": "1.1",
        "method": "GET",
        "target": "/apis/avid.iam/principals/123",
        [...]
      }
    }
    "method": "GET"
   },
  "paramSet": {
      "regionId": "123"
  }
}

Response
HTTP/1.1 200 OK
[...]
{
  "field1": "value1",
  "field2": "value2"
}
{
  "serviceType": "avid.iam",
  "serviceRealm": "global",
  "serviceVersion": 0,
  "op": "operation",
  "resultSet": {
    "body": {
      "encoding": "json",
      "data": {
        "field1": "value1",
        "field2": "value2"
      }
    }
  }
}