Learn how to configure API sources in Mage to extract data from RESTful and file-based endpoints. This guide covers authentication, pagination, response parsing, and advanced options available in Mage Pro.
Key | Description | Required or optional |
---|---|---|
url | API URL | Required |
method | GET or POST | Optional: GET is default |
query | URL query parameters. | Optional |
payload | When method is POST , this payload is used in the request body. | Optional |
headers | Request headers. | Optional |
response_parser | Parse the API response using dot notation. The final result must be an array. | Optional |
columns | If the final data returned from the API or from the response_parser is not a JSON object (e.g. it’s an array of strings or an array of array of strings), then you must define the columns. | Required (conditionally) |
separator | If the file extension is TSV, XLSX (or CSV with specific separator), then you can define it | Optional: ’,’ is default |
has_header | If the file extension is TSV, XLSX or CSV, and contains a header, then you can define it as True | Optional: False is default |
Key | Description | Default |
---|---|---|
encoding | Specifies the character encoding of the API response, such as "utf-8" or "iso-8859-1" . Use this to avoid decoding issues with special characters. | "utf-8" |
verify | Whether to verify SSL certificates for HTTPS requests. Disabling this is useful in internal or staging environments with self-signed certs. | True |
schema_discovery_sample_rows | The number of rows to sample when inferring the schema from API responses or files. A higher value improves schema accuracy for inconsistent data. | 100 |
force_all_columns_type | Forces all columns to be interpreted as a single type: "string" , "integer" , or "number" . Useful for normalizing schema across variable data sources. | None |
infer_schema_when_syncing | Enables schema inference at the time of sync (rather than preview), which is ideal for streaming or large file responses. | False |
streams
key. Each key in the streams
map represents a separate stream and can define its own URL, headers, parser, and pagination settings.
pagination_config
config block supports two types:
"cursor"
— follows a token provided in the API response (e.g., next_cursor
)"page"
— iterates using a page number or offset (e.g., page=1
, page=2
, …)Key | Description | Required |
---|---|---|
type | Must be "cursor" | Yes |
cursor_param | Name of the query parameter for the cursor (single cursor only) | Yes* |
cursor_params | List of query parameter names used for multiple cursors | Yes* |
initial_cursor_value | Initial value(s) to start pagination. Can be a string, list, or dictionary. | No |
next_cursor_path | Dot path(s) in response to extract next cursor. String for single, dictionary for multiple cursors. | Yes |
*EitherExample: Single cursorcursor_param
orcursor_params
must be provided. Ifcursor_params
is used,next_cursor_path
must be a dictionary mapping each param to its response path.
limit
parameter.
Key | Description | Required |
---|---|---|
type | "page" | ✅ |
param | Name of the query parameter that controls page or offset (e.g., "page" , "offset" ) | ✅ |
initial_value | Starting value for the page or offset | ✅ |
increment_by | Value to increase per request (e.g., 1 for pages, 100 for offset) | ✅ |
limit_param | Name of the limit parameter (e.g., "limit" , "per_page" ) | Optional |
limit_value | Limit value per page | Optional |
stop_condition.type | When to stop: "empty" (no results) or "max_value" (based on total pages) | ✅ |
stop_condition.max_value_path | (If using max_value ) Dot-notation path to total pages or count (e.g., meta.totalPages ) | Required for max_value |
response_parser
value of response.docs[0].author_display
,
the data that is extracted from the API’s response is:
columns
configuration value is required.
The final data is converted into a JSON object before being outputted to its destination:
response_parser
, the final data matches the exact response from the API.
Since each item in the final data is a JSON object, the columns
configuration value isn’t required.