REST API Connector can be used to fetch data from any REST endpoint. You can use the Flowrest API, which extends Unirest, to make rest calls.
Pagination With Page Number
Page Count Script
Record r = Flowrest.get(connection.getBaseUrl())
.basicAuth(connection.getUsername(), connection.getPassword())
.record();
return (r.count / 100) + 1; // this api returns 100 rows per page
Fetch Records Script
String url = "${connection.baseUrl}?page=${page}";
Record r = Flowrest.get(url)
.basicAuth(connection.username, connection.password)
.record();
return r.results; // same as r.get("results")
Sequential Pagination With Last Update Date
Page Count Script
return 1;
Fetch Records Script
String url;
if (scope.nextPageUrl != null) { // check if nextPageUrl is set in scope
url = scope.nextPageUrl; // incremental pages...
} else if (!offset.isInitialRun()) { // incremental runs, use the previous run's max last update date value from the offset store and pass to the api filter
url = "${connection.baseUrl}/api?limit=1000&q=lastUpdateDate gt ${offset.formatLastUpdateDateValue("yyyy-MM-dd'T'HH:mm:ssXXX")}";
} else {
url = "${connection.baseUrl}/api?limit=1000"; // first time call
}
Record r = Flowrest.get(url)
.basicAuth(connection.username, connection.password) // basic auth
.record(); // returns JSON Object as Record
scope.hasNextPage = r.next != null; // scope.hasNextPage must be set to true for this script to be invoked in loop
scope.nextPageUrl = r.next; // set the next page url to be used in the next run... see line 3 above
return r.results; // same as r.get("results")
scope is of type Record that can be used to store any variables to be re-used across multi-page requests
Paste a sample payload with either a single JSON Object or an Array of Objects
Click Generate Schema to auto-generate schema based on the sample payload
Verify and update the schema as necessary
The more accurate schema will result in less runtime data validation errors
