HTTP Adapter

The http adapter allows reading points from, or writing points to, an http endpoint.

read http

Read points by issuing an HTTP request and pushing the response into the Juttle flowgraph, with options:

read http -url url
          -method method
          -headers headers
          -body body
          -timeField timeField
          -rootPath rootPath
          -includeHeaders true/false
          -format csv/json/jsonl
          -followLinkHeader true/false
          -pageParam parameter
          -pageStart start
          -pageUnit unit
          -separator character
          -commentSymbol character
          -ignoreEmptyLines true/false
          -allowIncompleteLines true/false
Parameter Description Required?
-url URL to issue the HTTP request at Yes
-method HTTP method to use No; default: GET
-headers headers to attach to the HTTP request in the form { key1: "value1", ..., keyN: "valueN" } No; default: {}
-body body to send with the HTTP requests No; default: {}
-timeField The name of the field to use as the time field

The specified field will be renamed to timein the body of the HTTP request. If the points already contain a field called time that field is overwritten. This is useful when the source data contains a time field whose values are not valid time stamps.
No; defaults to keeping the timefield as is
-rootPath When the incoming data is JSON, use the specified path into the incoming object (expressed as field1.field2 to emit points No
-includeHeaders When set to true the headers from the response are appended to each of the points in the response No; default: false
-timeField Name of the field in the data which contains a valid timestamp No; defaults to time
-format HTTP data format, supports: csv jsonand jsonl No; defaults to HTTP response header Content-Type
-followLinkHeader When set to true the Web Linking feature is enabled which allows for automatic handling of paging as specified by the RFC5988. No; default false
-pageParam The name of the query parameter used to enable paging with multiple HTTP requests sent to the URL provided until response contains 0 records No; defaults to paging behavior disabled
-pageStart Initial value of the paging offset to set the pageParamto in the URL used to make the HTTP request. No; default: 0
-pageUnit <request|record> If set to request then each successive HTTP paging request increments the paging query parameter by one. If set to record then each successive request increments the paging query parameter by the number of records returned in the previous request. No; default: request
-pageField If set, then each successive HTTP paging request sets the paging query parameter to the last value of the field with name pageField Cannot be used with pageStartor pageUnit No
-separator When -format 'csv'is used, you can specify the separator between columns in a CSV file. No: defaults to ,
-commentSymbol When -format 'csv'is used, you can specify the comment character that prefixes comment lines in a CSV file. No: defaults to ,
-ignoreEmptyLines When -format 'csv'is used, you can skip empty lines in a CSV file. No: defaults to false
-allowIncompleteLines When -format 'csv'is used, you can allow for parsing of incomplete lines in a CSV file. No: defaults to false

Currently the read http adapter will automatically parse incoming data based off of the content-type header. Here are the currently supported content-types:

* `text/csv`: for [CSV](https://tools.ietf.org/html/rfc4180) data
* `application/json` for [JSON](https://tools.ietf.org/html/rfc7159) data
* `application/json` for [JSON lines](http://jsonlines.org/) data

Example

Example of how to hit the Github Issues API and retrieve all issues from the beginning of time on a specific repository in the correct chronological order:

/*
 * List issues on github 
 *
 * Input parameters:
 * 
 *  owner: repository owner, ie 'nodejs'
 *  repo: repository name, ie 'node'
 *  page: which of the n pages of issues to display, default: 1
 *  perPage: how many results per page to pull, default 10
 *  state: which state of the bugs you're interested in, default 'all'
 * 
 * For more information on the github Issues API, see: 
 *  https://developer.github.com/v3/issues/ 
 * 
 */

input owner: text -default 'nodejs';
input repo: text -default 'node';
input page: number -default 1;
input perPage: number -default 10;
input state: text -default 'all'; 

read http -url "https://api.github.com/repos/${owner}/${repo}/issues?page=${page}&per_page=${perPage}&state=${state}&direction=asc&sort=created"
          -headers { 'User-Agent': 'http-adapter-example' }
| keep title, created_at, state, number
| put time=Date.new(created_at)
| view text

Example

Example of how to hit the NPM download counts API and retrieve daily download counts for a given package. This uses the rootPath and timeField options:

/*
 * List issues on github 
 *
 * Input parameters:
 * 
 *  owner: repository owner, ie 'nodejs'
 *  repo: repository name, ie 'node'
 *  page: which of the n pages of issues to display, default: 1
 *  perPage: how many results per page to pull, default 10
 *  state: which state of the bugs you're interested in, default 'all'
 * 
 * For more information on the github Issues API, see: 
 *  https://developer.github.com/v3/issues/ 
 * 
 */

input owner: text -default 'nodejs';
input repo: text -default 'node';
input page: number -default 1;
input perPage: number -default 10;
input state: text -default 'all'; 

read http -url "https://api.github.com/repos/${owner}/${repo}/issues?page=${page}&per_page=${perPage}&state=${state}&direction=asc&sort=created"
          -headers { 'User-Agent': 'http-adapter-example' }
| keep title, created_at, state, number
| put time=Date.new(created_at)
| view text

Example

Example of how to access the Travis CI builds api and retrieve daily build counts for a given repository. This uses the -pageParam, -pageUnit=field, and -pageField options:

read http -url "https://api.travis-ci.org/repos/juttle/juttle/builds"
    -headers {
        'Authorization': 'token --TRAVIS-CI-AUTH-TOKEN-HERE--',
        'Accept': 'application/vnd.travis-ci.2+json',
        'User-Agent': 'Juttle TravisCI Testing'
    }
    -rootPath "builds"
    -timeField 'finished_at'
    -pageField "number"
    -pageParam "after_number"
| keep time, number, state
| view text

write http

Write points out of the Juttle flowgraph by making an HTTP request, with options:

write http -url url
           -method method
           -headers headers
           -maxLength length
Parameter Description Required?
-url URL to issue the HTTP request at Yes
-method HTTP method to use No; default: POST
-headers headers to attach to the HTTP request in the form { key1: "value1", ..., keyN: "valueN" } No; default: {}
-maxLength maximum payload length per HTTP request, as number of data points (not bytes)

If the number of data points out of the flowgraph exceeds maxLengththen multiple HTTP requests will be sent.
No, default: 1 (each data point out of the flowgraph becomes one HTTP request)

Each set of points passed along by the Juttle flowgraph will be placed into the body of an HTTP request, as defined by the specified options. The maxLength parameter allows to dictate how many points to push at most with each HTTP request. On failure, a warning will be displayed.

Example

Simple example of using the http adapter to create gists in GitHub:

/*
 * Create a Gist on Github
 *
 * Input parameters:
 * 
 *  oauthToken: required  
 * 
 * For more information on the github Issues API, see: 
 *  https://developer.github.com/v3/issues/ 
 * 
 * Usage:
 * > ./bin/juttle docs/examples/adapters/http_post_to_a_gist.juttle --input oauthToken=XXX
 * 
 */

input oauthToken: text -default 'FILL_ME_IN';

const date = :now:;

emit -limit 5
| put value=count()
| put description='Test Gist ${date} #${value}',
      public=true,
      files = {
        'main.juttle' : {
            'content': 'emit -limit 1 | put message "hello world"'
        } 
      }
| keep description, public, files
| write http -url "https://api.github.com/gists"
             -method 'POST'
             -headers {
                'User-Agent': 'http-adapter-example',
                'Authorization': 'token ${oauthToken}'
             }