HTTP Adapter

The http adapter allows reading points from, or writing points to, an http endpoint.

HTTP Adapter
- read http
- write http

read http

Read points by issuing an HTTP request and pushing the response into the Juttle flowgraph, with options:

read http -url url
          -method method
          -headers headers
          -body body
          -timeField timeField
          -rootPath rootPath
          -includeHeaders true/false
          -format csv/json/jsonl
          -followLinkHeader true/false
          -pageParam parameter
          -pageStart start
          -pageUnit unit
          -separator character
          -commentSymbol character
          -ignoreEmptyLines true/false
          -allowIncompleteLines true/false

Parameter	Description	Required?
`-url`	URL to issue the HTTP request at	Yes
`-method`	HTTP method to use	No; default: `GET`
`-headers`	headers to attach to the HTTP request in the form `{ key1: "value1", ..., keyN: "valueN" }`	No; default: `{}`
`-body`	body to send with the HTTP requests	No; default: `{}`
`-timeField`	The name of the field to use as the time field The specified field will be renamed to `time`in the body of the HTTP request. If the points already contain a field called `time` that field is overwritten. This is useful when the source data contains a time field whose values are not valid time stamps.	No; defaults to keeping the `time`field as is
`-rootPath`	When the incoming data is JSON, use the specified path into the incoming object (expressed as `field1.field2` to emit points	No
`-includeHeaders`	When set to true the headers from the response are appended to each of the points in the response	No; default: `false`
`-timeField`	Name of the field in the data which contains a valid timestamp	No; defaults to `time`
`-format`	HTTP data format, supports: `csv` `json`and `jsonl`	No; defaults to HTTP response header `Content-Type`
`-followLinkHeader`	When set to true the Web Linking feature is enabled which allows for automatic handling of paging as specified by the RFC5988.	No; default `false`
`-pageParam`	The name of the query parameter used to enable paging with multiple HTTP requests sent to the URL provided until response contains 0 records	No; defaults to paging behavior disabled
`-pageStart`	Initial value of the paging offset to set the `pageParam`to in the URL used to make the HTTP request.	No; default: `0`
`-pageUnit <request\|record>`	If set to `request` then each successive HTTP paging request increments the paging query parameter by one. If set to `record` then each successive request increments the paging query parameter by the number of records returned in the previous request.	No; default: `request`
`-pageField`	If set, then each successive HTTP paging request sets the paging query parameter to the last value of the field with name `pageField` Cannot be used with `pageStart`or `pageUnit`	No
`-separator`	When `-format 'csv'`is used, you can specify the separator between columns in a CSV file.	No: defaults to `,`
`-commentSymbol`	When `-format 'csv'`is used, you can specify the comment character that prefixes comment lines in a CSV file.	No: defaults to `,`
`-ignoreEmptyLines`	When `-format 'csv'`is used, you can skip empty lines in a CSV file.	No: defaults to `false`
`-allowIncompleteLines`	When `-format 'csv'`is used, you can allow for parsing of incomplete lines in a CSV file.	No: defaults to `false`

Currently the read http adapter will automatically parse incoming data based off of the content-type header. Here are the currently supported content-types:

* `text/csv`: for [CSV](https://tools.ietf.org/html/rfc4180) data
* `application/json` for [JSON](https://tools.ietf.org/html/rfc7159) data
* `application/json` for [JSON lines](http://jsonlines.org/) data

Example

Example of how to hit the Github Issues API and retrieve all issues from the beginning of time on a specific repository in the correct chronological order:

/*
 * List issues on github 
 *
 * Input parameters:
 * 
 *  owner: repository owner, ie 'nodejs'
 *  repo: repository name, ie 'node'
 *  page: which of the n pages of issues to display, default: 1
 *  perPage: how many results per page to pull, default 10
 *  state: which state of the bugs you're interested in, default 'all'
 * 
 * For more information on the github Issues API, see: 
 *  https://developer.github.com/v3/issues/ 
 * 
 */

input owner: text -default 'nodejs';
input repo: text -default 'node';
input page: number -default 1;
input perPage: number -default 10;
input state: text -default 'all'; 

read http -url "https://api.github.com/repos/${owner}/${repo}/issues?page=${page}&per_page=${perPage}&state=${state}&direction=asc&sort=created"
          -headers { 'User-Agent': 'http-adapter-example' }
| keep title, created_at, state, number
| put time=Date.new(created_at)
| view text

Example

Example of how to hit the NPM download counts API and retrieve daily download counts for a given package. This uses the rootPath and timeField options:

/*
 * List issues on github 
 *
 * Input parameters:
 * 
 *  owner: repository owner, ie 'nodejs'
 *  repo: repository name, ie 'node'
 *  page: which of the n pages of issues to display, default: 1
 *  perPage: how many results per page to pull, default 10
 *  state: which state of the bugs you're interested in, default 'all'
 * 
 * For more information on the github Issues API, see: 
 *  https://developer.github.com/v3/issues/ 
 * 
 */

input owner: text -default 'nodejs';
input repo: text -default 'node';
input page: number -default 1;
input perPage: number -default 10;
input state: text -default 'all'; 

read http -url "https://api.github.com/repos/${owner}/${repo}/issues?page=${page}&per_page=${perPage}&state=${state}&direction=asc&sort=created"
          -headers { 'User-Agent': 'http-adapter-example' }
| keep title, created_at, state, number
| put time=Date.new(created_at)
| view text

Example

Example of how to access the Travis CI builds api and retrieve daily build counts for a given repository. This uses the -pageParam, -pageUnit=field, and -pageField options:

read http -url "https://api.travis-ci.org/repos/juttle/juttle/builds"
    -headers {
        'Authorization': 'token --TRAVIS-CI-AUTH-TOKEN-HERE--',
        'Accept': 'application/vnd.travis-ci.2+json',
        'User-Agent': 'Juttle TravisCI Testing'
    }
    -rootPath "builds"
    -timeField 'finished_at'
    -pageField "number"
    -pageParam "after_number"
| keep time, number, state
| view text

write http

Write points out of the Juttle flowgraph by making an HTTP request, with options:

write http -url url
           -method method
           -headers headers
           -maxLength length

Parameter	Description	Required?
`-url`	URL to issue the HTTP request at	Yes
`-method`	HTTP method to use	No; default: `POST`
`-headers`	headers to attach to the HTTP request in the form `{ key1: "value1", ..., keyN: "valueN" }`	No; default: `{}`
`-maxLength`	maximum payload length per HTTP request, as number of data points (not bytes) If the number of data points out of the flowgraph exceeds `maxLength`then multiple HTTP requests will be sent.	No, default: 1 (each data point out of the flowgraph becomes one HTTP request)

Each set of points passed along by the Juttle flowgraph will be placed into the body of an HTTP request, as defined by the specified options. The maxLength parameter allows to dictate how many points to push at most with each HTTP request. On failure, a warning will be displayed.

Example

Simple example of using the http adapter to create gists in GitHub:

/*
 * Create a Gist on Github
 *
 * Input parameters:
 * 
 *  oauthToken: required  
 * 
 * For more information on the github Issues API, see: 
 *  https://developer.github.com/v3/issues/ 
 * 
 * Usage:
 * > ./bin/juttle docs/examples/adapters/http_post_to_a_gist.juttle --input oauthToken=XXX
 * 
 */

input oauthToken: text -default 'FILL_ME_IN';

const date = :now:;

emit -limit 5
| put value=count()
| put description='Test Gist ${date} #${value}',
      public=true,
      files = {
        'main.juttle' : {
            'content': 'emit -limit 1 | put message "hello world"'
        } 
      }
| keep description, public, files
| write http -url "https://api.github.com/gists"
             -method 'POST'
             -headers {
                'User-Agent': 'http-adapter-example',
                'Authorization': 'token ${oauthToken}'
             }