> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mage.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# API Source Configuration
> Learn how to configure API sources in Mage to extract data from RESTful and file-based endpoints. This guide covers authentication, pagination, response parsing, and advanced options available in Mage Pro.
Mage supports integrating with any RESTful or file-based API. This allows you to extract data from JSON, CSV, TSV, XLSX responses, with support for pagination, authentication, and schema inference.
📚 Sample public APIs for testing: [https://apipheny.io/free-api/](https://apipheny.io/free-api/)
## Basic Features (OSS)
### Configuration Fields
You must enter the following credentials when configuring this source:
| Key | Description | Required or optional |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- |
| `url` | API URL | Required |
| `method` | `GET` or `POST` | Optional: `GET` is default |
| `query` | URL query parameters. | Optional |
| `payload` | When method is `POST`, this payload is used in the request body. | Optional |
| `headers` | Request headers. | Optional |
| `response_parser` | Parse the API response using dot notation. The final result must be an array. | Optional |
| `columns` | If the final data returned from the API or from the `response_parser` is not a JSON object (e.g. it’s an array of strings or an array of array of strings), then you must define the columns. | Required (conditionally) |
| `separator` | If the file extension is TSV, XLSX (or CSV with specific separator), then you can define it | Optional: ',' is default |
| `has_header` | If the file extension is TSV, XLSX or CSV, and contains a header, then you can define it as True | Optional: False is default |
***
## Mage Pro Features
The following features are available only in [**Mage Pro**](https://cloud.mage.ai) and are designed for advanced API integrations at scale:
### Advanced Configuration (Mage Pro only)
These configuration parameters give you greater control over how data is fetched, parsed, and validated from APIs—especially useful for large files, custom encodings, or complex schemas.
| Key | Description | Default |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| `encoding` | Specifies the character encoding of the API response, such as `"utf-8"` or `"iso-8859-1"`. Use this to avoid decoding issues with special characters. | `"utf-8"` |
| `verify` | Whether to verify SSL certificates for HTTPS requests. Disabling this is useful in internal or staging environments with self-signed certs. | `True` |
| `schema_discovery_sample_rows` | The number of rows to sample when inferring the schema from API responses or files. A higher value improves schema accuracy for inconsistent data. | `100` |
| `force_all_columns_type` | Forces all columns to be interpreted as a single type: `"string"`, `"integer"`, or `"number"`. Useful for normalizing schema across variable data sources. | *None* |
| `infer_schema_when_syncing` | Enables schema inference at the time of sync (rather than preview), which is ideal for streaming or large file responses. | `False` |
These options enhance your ability to handle diverse data formats and edge cases in production environments, giving you full control over data ingestion quality and reliability.
### Multi-Stream Configuration (Mage Pro only)
Mage Pro supports defining multiple independent API configs using the `streams` key. Each key in the `streams` map represents a separate stream and can define its own URL, headers, parser, and pagination settings.
```yaml theme={"system"}
streams:
users:
url: https://api.example.com/users
headers:
x-api-key: user-key
response_parser: data.users
orders:
url: https://api.example.com/orders
headers:
x-api-key: order-key
response_parser: data.orders
```
Each stream entry can independently use any configuration field supported in the single-stream format.
### Pagination Configuration
To sync paginated API responses, the `pagination_config` config block supports two types:
* **`"cursor"`** — follows a token provided in the API response (e.g., `next_cursor`)
* **`"page"`** — iterates using a page number or offset (e.g., `page=1`, `page=2`, …)
***
#### Cursor-Based Pagination
Use when the API returns a **next cursor/token** in the response body.
| Key | Description | Required |
| ---------------------- | --------------------------------------------------------------------------------------------------- | -------- |
| `type` | Must be `"cursor"` | Yes |
| `cursor_param` | Name of the query parameter for the cursor (single cursor only) | Yes\* |
| `cursor_params` | List of query parameter names used for multiple cursors | Yes\* |
| `initial_cursor_value` | Initial value(s) to start pagination. Can be a string, list, or dictionary. | No |
| `next_cursor_path` | Dot path(s) in response to extract next cursor. String for single, dictionary for multiple cursors. | Yes |
> \*Either `cursor_param` or `cursor_params` must be provided.
> If `cursor_params` is used, `next_cursor_path` must be a dictionary mapping each param to its response path.
**Example: Single cursor**
```yaml theme={"system"}
pagination_config:
type: cursor
cursor_param: next
initial_cursor_value: null
next_cursor_path: meta.next_cursor
```
**Example: Multiple cursors**
```yaml theme={"system"}
pagination_config:
type: cursor
cursor_params:
- start
- end
initial_cursor_value:
start: null
end: null
next_cursor_path:
start: meta.start_cursor
end: meta.end_cursor
```
#### Page-Based Pagination
Use when the API accepts **page numbers** or **offsets** and optionally supports a `limit` parameter.
| Key | Description | Required |
| ------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------ |
| `type` | `"page"` | ✅ |
| `param` | Name of the query parameter that controls page or offset (e.g., `"page"`, `"offset"`) | ✅ |
| `initial_value` | Starting value for the page or offset | ✅ |
| `increment_by` | Value to increase per request (e.g., `1` for pages, `100` for offset) | ✅ |
| `limit_param` | Name of the limit parameter (e.g., `"limit"`, `"per_page"`) | Optional |
| `limit_value` | Limit value per page | Optional |
| `stop_condition.type` | When to stop: `"empty"` (no results) or `"max_value"` (based on total pages) | ✅ |
| `stop_condition.max_value_path` | (If using `max_value`) Dot-notation path to total pages or count (e.g., `meta.totalPages`) | Required for `max_value` |
***
**Example: Page + Total Pages (stop when last page reached)**
```json theme={"system"}
pagination_config:
type: page
param: page
initial_value: 1
increment_by: 1
stop_condition:
type: max_value
max_value_path: total_pages
```
**Example: Offset + Limit (stop when response is empty)**
```json theme={"system"}
pagination_config:
type: page
param: offset
initial_value: 0
increment_by: 100
limit_param: limit
limit_value: 100
stop_condition:
type: empty
```
## Other Example Configs
### Example GET API 1
```json theme={"system"}
url: https://api.plos.org/search
query:
q: "title:DNA"
headers:
Content-Type: application/json
response_parser: response.docs[0].author_display
columns:
- author_display
```
The response from the above endpoint is:
```json theme={"system"}
{
"response": {
"numFound": 5669,
"start": 0,
"maxScore": 6.7217336,
"docs": [
{
"id": "10.1371/journal.pone.0000290",
"journal": "PLoS ONE",
"eissn": "1932-6203",
"publication_date": "2007-03-14T00:00:00Z",
"article_type": "Research Article",
"author_display": [
"Rayna I. Kraeva",
"Dragomir B. Krastev",
"Assen Roguev",
"Anna Ivanova",
"Marina N. Nedelcheva-Veleva",
"Stoyno S. Stoynov"
],
"abstract": [
"Nucleic acids, due to their structural and chemical properties, can form double-stranded secondary structures that assist the transfer of genetic information and can modulate gene expression. ",
"However, the nucleotide sequence alone is insufficient in explaining phenomena like intron-exon recognition during RNA processing. This raises the question whether nucleic acids are endowed with other attributes that can contribute to their biological functions. ",
"In this work, we present a calculation of thermodynamic stability of DNA/DNA and mRNA/DNA duplexes across the genomes of four species in the genus Saccharomyces by nearest-neighbor method. ",
"The results show that coding regions are more thermodynamically stable than introns, 3′-untranslated regions and intergenic sequences. Furthermore, open reading frames have more stable sense mRNA/DNA duplexes than the potential antisense duplexes, a property that can aid gene discovery. ",
"The lower stability of the DNA/DNA and mRNA/DNA duplexes of 3′-untranslated regions and the higher stability of genes correlates with increased mRNA level. These results suggest that the thermodynamic stability of DNA/DNA and mRNA/DNA duplexes affects mRNA transcription."
],
"title_display": "Stability of mRNA/DNA and DNA/DNA Duplexes Affects mRNA Transcription",
"score": 6.7217336
}
]
}
}
```
However, with the `response_parser` value of `response.docs[0].author_display`,
the data that is extracted from the API’s response is:
```json theme={"system"}
[
"Rayna I. Kraeva",
"Dragomir B. Krastev",
"Assen Roguev",
"Anna Ivanova",
"Marina N. Nedelcheva-Veleva",
"Stoyno S. Stoynov"
]
```
Since each item in the final data is not a JSON object, the `columns` configuration value is required.
The final data is converted into a JSON object before being outputted to its destination:
```json theme={"system"}
[
{
"author_display": "Rayna I. Kraeva"
},
{
"author_display": "Dragomir B. Krastev"
},
{
"author_display": "Assen Roguev"
},
{
"author_display": "Anna Ivanova"
},
{
"author_display": "Marina N. Nedelcheva-Veleva"
},
{
"author_display": "Stoyno S. Stoynov"
}
]
```
### Example GET API 2
```json theme={"system"}
url: https://api.coingecko.com/api/v3/coins/markets
query:
vs_currency: usd
```
The response from the above endpoint is:
```json theme={"system"}
[
{
"id": "bitcoin",
"symbol": "btc",
"name": "Bitcoin",
"image": "https://assets.coingecko.com/coins/images/1/large/bitcoin.png?1547033579",
"current_price": 17154.23,
"market_cap": 329331527834,
"market_cap_rank": 1,
"fully_diluted_valuation": 359638803581,
"total_volume": 14339246353,
"high_24h": 17227.56,
"low_24h": 17107.75,
"price_change_24h": 18.57,
"price_change_percentage_24h": 0.10839,
"market_cap_change_24h": -425260465.15130615,
"market_cap_change_percentage_24h": -0.12896,
"circulating_supply": 19230300.0,
"total_supply": 21000000.0,
"max_supply": 21000000.0,
"ath": 69045,
"ath_change_percentage": -75.16662,
"ath_date": "2021-11-10T14:24:11.849Z",
"atl": 67.81,
"atl_change_percentage": 25185.95387,
"atl_date": "2013-07-06T00:00:00.000Z",
"roi": null,
"last_updated": "2022-12-11T00:32:01.695Z"
}
]
```
Because there is no `response_parser`, the final data matches the exact response from the API.
Since each item in the final data is a JSON object, the `columns` configuration value isn’t required.
### Example POST API
```yaml theme={"system"}
url: https://api.something.com/users
method: POST
payload:
user:
first_name: Urza
power: 10
headers:
"Content-Type": "application/json"
token: abc123
response_parser: "user"
```
### Example XLSX or CSV
```yaml theme={"system"}
url: https://docs.google.com/spreadsheets/d/e/2PACX-1vTLcLUBAJAWf-8NQSjlbB3E4LR6DWk5QIZC-KtRb1j2CXXcgY6mE6vOJAW8PoJ1BAOgjXYpE4tY1LAD/pub?output=xlsx
method: GET
has_header: True
```
The response from the above endpoint is:
```
teste,first_name,second_name,email
1,Sadella,Tythacott,stythacott0@sina.com.cn
2,Melessa,Flaune,mflaune1@si.edu
3,Caroljean,Filipowicz,cfilipowicz2@guardian.co.uk
4,Doll,Wannan,dwannan3@people.com.cn
5,Nancy,Giraudy,ngiraudy4@pagesperso-orange.fr
6,Dominic,Bimson,dbimson5@vinaora.com
7,Kikelia,Bishopp,kbishopp6@cdc.gov
8,Andrus,Pomfrett,apomfrett7@wikipedia.org
9,Wildon,Fillingham,wfillingham8@google.co.uk
10,Alfonse,Leechman,aleechman9@jalbum.net
11,Phil,Emblem,pemblema@opera.com
12,Eyde,Brewer,ebrewerb@istockphoto.com
....
```
Data can also be checked by clicking on the "Load Sample data" button.