Avid | AI Media Analytics API

Providers for AI Media Analytics are services that need to utilize the Avid Connector API.

The provider needs to implement the following operations:

  • getSupportedTypes
  • startJob
  • getJob
  • getResult
  • getResultForType
  • delete
  • search
  • cancel

All operations need to be exposed to Upstream and can return the following HTTP status codes:

  • 200
  • 500
  • 401
  • 404

Clients need to use Platform Authentication to obtain a valid access token.

Analytic Types

The analytic types determine the supported analysis types of the provider.
A provider can define its own analytic types, however with the 2018.9 release the Avid | AI Media Analytics product supports following types:

  • Annotations - annotation of the visual content
  • Audio Effects - detects audio effects, such as silence
  • Brands - (Brands) detect brands
  • Content Moderation - detects adult and racial content
  • Emotions - (Emotions) detect emotions
  • Face Recognition - detects faces in the video
  • Frame Patterns - detects frame patterns, such as “black frames”
  • Keywords - detects keywords for the audio or video content
  • Metadata - provides asset metadata, such as the main language of the video
  • OCR - (Optical Character Recognition) detects and extracts text that is displayed in the video
  • People - (People) occurances of named people
  • Sentiments - detects sentiment levels (positive, neutral, or negative)
  • Scenes - detects scene changes
  • STT - (Speech To Text) creates transcripts of spoken words
  • Topics - (Topics) detect topics

Data Model

The results of an analytic job can differ per provider and per type.
The Avid | AI Media Analytics 2018.9 contains predefined attributes & layers for Asset Management assets that can store the results of an analytics job.

A detailed description of the data model can be found here.

getSupportedTypes Operation

The getSupportedTypes operation returns the analytic types grouped by media type that this media analytics provider supports.

Upstream Route: /analytics/types

Content-Type: application/json

Accept: application/json

HTTP Method: GET

{
      "Video": [
        "Annotations",
        "AudioEffects",
        "Brands",
        "ContentModeration",
        "Emotions",
        "FaceDetection",
        "FramePatterns",
        "Keywords",
        "Metadata",
        "OCR",
        "People",
        "Scenes",
        "Sentiments",
        "STT",
        "Topics"
      ],
      "Audio": [
        "AudioEffects",
        "Brands",
        "ContentModeration",
        "Keywords",
        "Metadata",
        "People",
        "Sentiments",
        "STT",
        "Topics"
      ]
}

Job Operations

The Job operations provide access to the analytic jobs.

startJob Operation

The start job operation starts a new Media Analytics Job based on the passed jobSpec and returns the jobId.

Upstream Route: /analytics

Content-Type: application/json

Accept: application/json

HTTP Method: POST

jobSpec:

{
    "sourceUrl": "\\\\<server>\\Folder\\SubFolder\\SomeFile.mp4",
    "language": "en",
    "types": [
        "Annotations",
        "AudioEffects",
        "FaceDetection",
        "FramePatterns",
        "STT",
        "Metadata",
        "OCR",
        "Scenes",
        "Sentiments"
    ],
    "parameters": {
        "editRate": "30000/1001"
    }
}
  • sourceUrl - The sourceUrl to the file, this can be a UNC path or any other URI that the provider supports
  • language - The language that is used for analysis (e.g. for speech to text)
  • types - The list of analysis types for the analysis job
  • parameters - A list of additional parameters for the analysis job

getJob Operation

The getJob operation returns the analytics job by the given id.

Upstream Route: /analytics/{id}/state

Content-Type: application/json

Accept: application/json

HTTP Method: GET

Result:

{
    "id": "37c792e6-97cd-4319-9529-c37e54e282f9",
    "state": "error",
    "message": "Some status message",
    "error": "Some error message",
    "progress": 55.78
}

The job state can be one of following values:

  • undefined - The job state is unknown
  • pending - The job is pending
  • uploading - The job is uploading the file to a remote storage
  • uploaded - The job uploaded the file to a remote storage
  • processing - The job is processing the file
  • finished - The job finished successfully
  • error - The job failed
  • canceled - The job was cancelled

delete Operation

The delete operation deletes the analytics job with the given id.

Upstream Route: /analytics/{id}

Content-Type: application/json

Accept: application/json

HTTP Method: DELETE

search Operation

The search operation returns the analytic jobs based on the AnalyticsJobsFilter.

Upstream Route: /analytics/search

HTTP Method: POST

AnalyticJobFilter:

{
    "skip": 0,
    "pageSize": 10
}

Result:

{
    "jobs": [{
        "id": "afa30b23-ba66-49f3-91de-a731f1d83901",
        "state": "error",
        "message": null,
        "error": "Failed to analyze file",
        "progress": 55.78
    },
    {
        "id": "72dec2f9-ed52-4d7f-8f5c-48ba83d2e70a",
        "state": "uploading",
        "message": "Uploading file ...",
        "error": null,
        "progress": 10.0
    },
    {
        "id": "f3aaf563-c348-414c-856d-a228f8320f33",
        "state": "finished",
        "message": null,
        "error": null,
        "progress": 100
    }],
    "nextPage": {
        "skip": 0,
        "pageSize": 10,
        "done": true
    }
}

cancel Operation

The cancel operation aborts a running analytics job.

Upstream Route: /analytics/{id}/cancel

Content-Type: application/json

Accept: application/json

HTTP Method: POST 

Analytic Results

Analytic results are in CTMS compatible format.

A result consist of

  • type - The analytic type of this result
  • dataType - The CTMS resource for the result (e.g. aa:time-based, aa:asset)
  • data - The json payload for the CTMS API endpoint that matches the resource given as dataType

getResult Operation

The getResult operation returns all results of the analytics job for the given id.

Upstream Route: /analytics/{id}

Content-Type: application/json

Accept: application/json

HTTP Method: GET

Result:

{
    "id": "37c792e6-97cd-4319-9529-c37e54e282f9",
    "results": [{
        "type": "Sentiments",
        "dataType": "aa:time-based",
        "data": {
            "editRate": "30000/1001",
            "dropFrame": false,
            "layers": [{
                "name": "MAN_Sentiments",
                "segments": [{
                    "id": "1",
                    "start": 0,
                    "duration": 1027,
                    "attributes": [{
                        "name": "MAN_Sentiment_Type",
                        "value": "Neutral"
                    }]
                },
                {
                    "id": "2",
                    "start": 1027,
                    "duration": 280,
                    "attributes": [{
                        "name": "MAN_Sentiment_Type",
                        "value": "Neutral"
                    }]
                }]
            }]
        }
    },
    {
        "type": "Metadata",
        "dataType": "aa:asset",
        "data": {
            "base": {
                
            },
            "common": {
                
            },
            "_links": {
                "self": {
                    "href": "http://dummy"
                },
                "aa:attributes": {
                    "href": "http://dummy"
                }
            },
            "_embedded": {
                "aa:attributes": {
                    "attributes": [{
                        "name": "MAN_IsRacialContent",
                        "value": false
                    },
                    {
                        "name": "MAN_Topics",
                        "value": [{
                            "attributes": [{
                                "name": "MAN_Topic_Name",
                                "value": "anthony dragon"
                            },
                            {
                                "name": "TopicRank",
                                "value": 4.0
                            }]
                        }]
                    },
                    {
                        "name": "MAN_Persons",
                        "value": [{
                            "attributes": [{
                                "name": "MAN_Person_Name",
                                "value": "Speaker #0"
                            },
                            {
                                "name": "MAN_Person_Title",
                                "value": null
                            },
                            {
                                "name": "MAN_Person_Description",
                                "value": null
                            },
                            {
                                "name": "MAN_ImageUrl",
                                "value": null
                            }]
                        },
                        {
                            "attributes": [{
                                "name": "MAN_Person_Name",
                                "value": "Speaker #1"
                            },
                            {
                                "name": "MAN_Person_Title",
                                "value": null
                            },
                            {
                                "name": "MAN_Person_Description",
                                "value": null
                            },
                            {
                                "name": "MAN_ImageUrl",
                                "value": null
                            }]
                        }]
                    }],
                    "_links": {
                        "self": {
                            "href": "http://dummy"
                        }
                    }
                }
            }
        }
    }]
}

getResultForType Operation

The getResultForType operation returns the result for a specific analytic type.

Upstream Route: /analytics/{id}/results/{type}

Content-Type: application/json

Accept:: application/json

HTTP Method: GET

Result:

{
    "type": "Sentiments",
    "dataType": "aa:time-based",
    "data": {
        "editRate": "30000/1001",
        "dropFrame": false,
        "layers": [{
            "name": "MAN_Sentiments",
            "segments": [{
                "id": "1",
                "start": 0,
                "duration": 1027,
                "attributes": [{
                    "name": "MAN_Sentiment_Type",
                    "value": "Neutral"
                }]
            },
            {
                "id": "2",
                "start": 1027,
                "duration": 280,
                "attributes": [{
                    "name": "MAN_Sentiment_Type",
                    "value": "Neutral"
                }]
            }]
        }]
    }
}