API Source Configuration
Learn how to configure API sources in Mage to extract data from RESTful and file-based endpoints. This guide covers authentication, pagination, response parsing, and advanced options available in Mage Pro.
Mage supports integrating with any RESTful or file-based API. This allows you to extract data from JSON, CSV, TSV, XLSX responses, with support for pagination, authentication, and schema inference.
📚 Sample public APIs for testing: https://apipheny.io/free-api/
Basic Features (OSS)
Configuration Fields
You must enter the following credentials when configuring this source:
| Key | Description | Required or optional |
|---|---|---|
url | API URL | Required |
method | GET or POST | Optional: GET is default |
query | URL query parameters. | Optional |
payload | When method is POST, this payload is used in the request body. | Optional |
headers | Request headers. | Optional |
response_parser | Parse the API response using dot notation. The final result must be an array. | Optional |
columns | If the final data returned from the API or from the response_parser is not a JSON object (e.g. it’s an array of strings or an array of array of strings), then you must define the columns. | Required (conditionally) |
separator | If the file extension is TSV, XLSX (or CSV with specific separator), then you can define it | Optional: ’,’ is default |
has_header | If the file extension is TSV, XLSX or CSV, and contains a header, then you can define it as True | Optional: False is default |
Mage Pro Features
The following features are available only in Mage Pro and are designed for advanced API integrations at scale:
Advanced Configuration (Mage Pro only)
These configuration parameters give you greater control over how data is fetched, parsed, and validated from APIs—especially useful for large files, custom encodings, or complex schemas.
| Key | Description | Default |
|---|---|---|
encoding | Specifies the character encoding of the API response, such as "utf-8" or "iso-8859-1". Use this to avoid decoding issues with special characters. | "utf-8" |
verify | Whether to verify SSL certificates for HTTPS requests. Disabling this is useful in internal or staging environments with self-signed certs. | True |
schema_discovery_sample_rows | The number of rows to sample when inferring the schema from API responses or files. A higher value improves schema accuracy for inconsistent data. | 100 |
force_all_columns_type | Forces all columns to be interpreted as a single type: "string", "integer", or "number". Useful for normalizing schema across variable data sources. | None |
infer_schema_when_syncing | Enables schema inference at the time of sync (rather than preview), which is ideal for streaming or large file responses. | False |
These options enhance your ability to handle diverse data formats and edge cases in production environments, giving you full control over data ingestion quality and reliability.
Pagination Configuration
To sync paginated API responses, the pagination_config config block supports two types:
"cursor"— follows a token provided in the API response (e.g.,next_cursor)"page"— iterates using a page number or offset (e.g.,page=1,page=2, …)
🔁 Cursor-Based Pagination
Use when the API returns a next cursor/token in the response body.
| Key | Description | Required |
|---|---|---|
type | "cursor" | ✅ |
cursor_param | Name of the query parameter to set the cursor/token | ✅ |
initial_cursor_value | Initial cursor value to use in the first request (e.g., "*" or null) | ✅ |
next_cursor_path | Dot-notation path in the response to extract the next cursor | ✅ |
Example
🔢 Page-Based Pagination
Use when the API accepts page numbers or offsets and optionally supports a limit parameter.
| Key | Description | Required |
|---|---|---|
type | "page" | ✅ |
param | Name of the query parameter that controls page or offset (e.g., "page", "offset") | ✅ |
initial_value | Starting value for the page or offset | ✅ |
increment_by | Value to increase per request (e.g., 1 for pages, 100 for offset) | ✅ |
limit_param | Name of the limit parameter (e.g., "limit", "per_page") | Optional |
limit_value | Limit value per page | Optional |
stop_condition.type | When to stop: "empty" (no results) or "max_value" (based on total pages) | ✅ |
stop_condition.max_value_path | (If using max_value) Dot-notation path to total pages or count (e.g., meta.totalPages) | Required for max_value |
Example: Page + Total Pages (stop when last page reached)
Example: Offset + Limit (stop when response is empty)
Other Example Configs
Example GET API 1
The response from the above endpoint is:
However, with the response_parser value of response.docs[0].author_display,
the data that is extracted from the API’s response is:
Since each item in the final data is not a JSON object, the columns configuration value is required.
The final data is converted into a JSON object before being outputted to its destination:
Example GET API 2
The response from the above endpoint is:
Because there is no response_parser, the final data matches the exact response from the API.
Since each item in the final data is a JSON object, the columns configuration value isn’t required.
Example POST API
Example XLSX or CSV
The response from the above endpoint is:
Data can also be checked by clicking on the “Load Sample data” button.