1.x

HTTP API Collector

Collect from REST APIs

The HTTP API collector allows you to collect from arbitrary REST APIs while using authentication. Additionally, the API collector supports paginated APIs via go templating.

Configuration Parameters

General

KeyDescription

URL

The Target URL for Calyptia Core to pull data from. This must be an absolute URL starting with HTTP or HTTPS, for example https://example.org/pages'

Header

HTTP headers used when calling the HTTP API. Multiple headers can be set.

Pull Interval

The amount of time between each API call.

Split Records

Boolean value on whether to split based on JSON array or JSON response

Advanced

KeyDescription

Body

Optional Request body E.g. {"custom":"data"}

Request Timeout

The amount of time between requests, default 10s

Continue Scrape on HTTP Error

Option on whether to continue upon recieving a >400 HTTP Error Response Code

Max Response Bytes

Option to limit the amount of bytes to read from the response body.

Template

Golang template that applies over the record. This include statusCode (int), headers (http.Header), and body which can be any and index which is an increasing number for the request being made. All template must be inserted within quotes

Stop Template

Golang Template that evaluates to a boolean value that tells the plugin when to stop

Next URL Template

An optional golang template you can use to modify the original request URL. This is useful when advanced or iterating over a paginated API. https://example.org/pages?offset={{.index}}

Next Body Template

An optional golang template you can use to modify the original request body

Next Headers Template

Optional golang template used to modufy the original request headers

OAuth2

KeyDescription

OAuth2 Client ID

Optional OAuth2 Client ID. You need to at least pass the client ID, secret and token URL to enable OAuth2, this uses the client_credentials flow

OAuth2 Client Secret

Optional OAuth2 client secret

OAuth2 Token URL

Optional OAuth2 token URL in the formal of HTTP or HTTPS, E.g. https://example.org/oauth2/token

OAuth2 Scope

Optional list of additional scopes for OAuth2 separated by space. E.g. "foo bar"

OAuth2 Scopes Seperator

Additional parameter to pass during OAuth2 that seperates scopes specified in OAuth2 Scopes

OAuth2 Endpoint Params

Optional additional parameters to add during OAuth2. The format is a URL query string, E.g. foo=bar&bar=qux

Go Templating

One of the main benefits of the HTTP API collector is the ability to perform custom logic on top of the HTTP API Collection that you are performing. For example, you can use the next_url_template to continue scanning a paginated API or you can add a custom loop for check a series of offsets A helpful list of go template commands can be found here: (http://masterminds.github.io/sprig/)

Tips and Tricks

When performing a pull from an API for the first time, the exploration of format of the API response is best done with curl or the API documentation.

Example of Okta Syslog Log API

The following is an example of Okta System Log Response, a paginated API and how to use the HTTP API templating to continue through the records

https://developer.okta.com/docs/reference/api/system-log/

Okta System Log Response
[
    {
        "version": "0",
        "severity": "INFO",
        "client": {
            "zone": "OFF_NETWORK",
            "device": "Unknown",
            "userAgent": {
                "os": "Unknown",
                "browser": "UNKNOWN",
                "rawUserAgent": "UNKNOWN-DOWNLOAD"
            },
            "ipAddress": "12.97.85.90"
        },
        "actor": {
            "id": "00u1qw1mqitPHM8AJ0g7",
            "type": "User",
            "alternateId": "admin@example.com",
            "displayName": "John Doe"
        },
        "outcome": {
            "result": "SUCCESS"
        },
        "uuid": "f790999f-fe87-467a-9880-6982a583986c",
        "published": "2017-09-31T22:23:07.777Z",
        "eventType": "user.session.start",
        "displayMessage": "User login to Okta",
        "transaction": {
            "type": "WEB",
            "id": "V04Oy4ubUOc5UuG6s9DyNQAABtc"
        },
        "debugContext": {
            "debugData": {
                "requestUri": "/login/do-login"
            }
        },
        "legacyEventType": "core.user_auth.login_success",
        "authenticationContext": {
            "authenticationStep": 0,
            "externalSessionId": "1013FfF-DKQSvCI4RVXChzX-w"
        }
    }
]

The response is a JSON array with objects inside. We need to send these objects independently through Calyptia Core pipelines so we will also enable the Split Records feature to true

Advanced Configuration within Calyptia Core Pipeline

[INPUT]
    Name              http_loader
    url               https://{yourOktaDomain}/api/v1/logs
    next_url_template {{nextLink .headers}}
    headers           Authorization: SSWS {yourOktaAPIToken}
    split_records     true
    template          {{toRawJson .body}}
    stop_template     false

Next URL Template

https://developer.okta.com/docs/reference/api/system-log/#next-link-response-header

The Next URL Template is a go-template that is used to build a new URL. In this case, the Okta API recommends using the response header Link rel=next to know which URL to use to advance to the next page. nextLink is a header function within go templates that takes that rel=next from the headers.

Split Records

As the response is a JSON array we want to split this into multiple records allowing us to perform processing on each one indepedently. To do this we use the template function toRawJSON function.

Stop Template

This parameter is used to tell the Calyptia Core pipeline when to stop fetching new fata. In this case we plan to continue tailing throughout time so we can pass a boolean value of false but we could also pass a go-template such as {{empty (nextLink .header)}} which would stop when a Link rel=next is no longer visible in the headers

Additional Go Template resources

PreviousHTTPNextSyslog

Last updated