HTTP API Collector

Collect from REST APIs

The HTTP API collector allows you to collect from arbitrary REST APIs while using authentication. Additionally, the API collector supports paginated APIs via go templating.

Configuration Parameters

General

Key	Description
URL	The Target URL for Calyptia Core to pull data from. This must be an absolute URL starting with HTTP or HTTPS, for example https://example.org/pages'
Header	HTTP headers used when calling the HTTP API. Multiple headers can be set.
Pull Interval	The amount of time between each API call.
Split Records	Boolean value on whether to split based on JSON array or JSON response

Advanced

Key	Description
Body	Optional Request body E.g. {"custom":"data"}
Request Timeout	The amount of time between requests, default 10s
Continue Scrape on HTTP Error	Option on whether to continue upon recieving a >400 HTTP Error Response Code
Max Response Bytes	Option to limit the amount of bytes to read from the response body.
Template	Golang template that applies over the record. This include statusCode (int), headers (http.Header), and body which can be any and index which is an increasing number for the request being made. All template must be inserted within quotes
Stop Template	Golang Template that evaluates to a boolean value that tells the plugin when to stop
Next URL Template	An optional golang template you can use to modify the original request URL. This is useful when advanced or iterating over a paginated API. https://example.org/pages?offset={{.index}}
Next Body Template	An optional golang template you can use to modify the original request body
Next Headers Template	Optional golang template used to modufy the original request headers

OAuth2

Key	Description
OAuth2 Client ID	Optional OAuth2 Client ID. You need to at least pass the client ID, secret and token URL to enable OAuth2, this uses the client_credentials flow
OAuth2 Client Secret	Optional OAuth2 client secret
OAuth2 Token URL	Optional OAuth2 token URL in the formal of HTTP or HTTPS, E.g. https://example.org/oauth2/token
OAuth2 Scope	Optional list of additional scopes for OAuth2 separated by space. E.g. "foo bar"
OAuth2 Scopes Seperator	Additional parameter to pass during OAuth2 that seperates scopes specified in OAuth2 Scopes
OAuth2 Endpoint Params	Optional additional parameters to add during OAuth2. The format is a URL query string, E.g. foo=bar&bar=qux

Go Templating

One of the main benefits of the HTTP API collector is the ability to perform custom logic on top of the HTTP API Collection that you are performing. For example, you can use the next_url_template to continue scanning a paginated API or you can add a custom loop for check a series of offsets A helpful list of go template commands can be found here: (http://masterminds.github.io/sprig/)

Tips and Tricks

When performing a pull from an API for the first time, the exploration of format of the API response is best done with curl or the API documentation.

Example of Okta Syslog Log API

The following is an example of Okta System Log Response, a paginated API and how to use the HTTP API templating to continue through the records

https://developer.okta.com/docs/reference/api/system-log/

Okta System Log Response

[
    {
        "version": "0",
        "severity": "INFO",
        "client": {
            "zone": "OFF_NETWORK",
            "device": "Unknown",
            "userAgent": {
                "os": "Unknown",
                "browser": "UNKNOWN",
                "rawUserAgent": "UNKNOWN-DOWNLOAD"
            },
            "ipAddress": "12.97.85.90"
        },
        "actor": {
            "id": "00u1qw1mqitPHM8AJ0g7",
            "type": "User",
            "alternateId": "[e-mail]",
            "displayName": "John Doe"
        },
        "outcome": {
            "result": "SUCCESS"
        },
        "uuid": "f790999f-fe87-467a-9880-6982a583986c",
        "published": "2017-09-31T22:23:07.777Z",
        "eventType": "user.session.start",
        "displayMessage": "User login to Okta",
        "transaction": {
            "type": "WEB",
            "id": "V04Oy4ubUOc5UuG6s9DyNQAABtc"
        },
        "debugContext": {
            "debugData": {
                "requestUri": "/login/do-login"
            }
        },
        "legacyEventType": "core.user_auth.login_success",
        "authenticationContext": {
            "authenticationStep": 0,
            "externalSessionId": "1013FfF-DKQSvCI4RVXChzX-w"
        }
    }
]

The response is a JSON array with objects inside. We need to send these objects independently through Calyptia Core pipelines so we will also enable the Split Records feature to true

Advanced Configuration within Calyptia Core Pipeline

[INPUT]
    Name              http_loader
    url               https://{yourOktaDomain}/api/v1/logs
    next_url_template {{nextLink .headers}}
    headers           Authorization: SSWS {yourOktaAPIToken}
    split_records     true
    template          {{toRawJson .body}}
    stop_template     false

Next URL Template

https://developer.okta.com/docs/reference/api/system-log/#next-link-response-header

The Next URL Template is a go-template that is used to build a new URL. In this case, the Okta API recommends using the response header Link rel=next to know which URL to use to advance to the next page. nextLink is a header function within go templates that takes that rel=next from the headers.

Split Records

As the response is a JSON array we want to split this into multiple records allowing us to perform processing on each one indepedently. To do this we use the template function toRawJSON function.

Stop Template

This parameter is used to tell the Calyptia Core pipeline when to stop fetching new fata. In this case we plan to continue tailing throughout time so we can pass a boolean value of false but we could also pass a go-template such as {{empty (nextLink .header)}} which would stop when a Link rel=next is no longer visible in the headers

Additional Go Template resources

• https://blog.gopheracademy.com/advent-2017/using-go-templates/

• https://philipptanlak.com/mastering-html-templates-in-go-the-fundamentals/

• https://developer.hashicorp.com/nomad/tutorials/templates/go-template-syntax

• http://masterminds.github.io/sprig/