Data integration
Airbyte in Mage
Data integration
Airbyte in Mage
Trigger a connection sync in Airbyte.
Configuration
Here are the following keyword arguments that can be used to configure Airbyte:
| Keyword argument | Description | Default value |
|---|---|---|
api_version | API version | v1 |
host | Airbyte server host | host.docker.internal |
password | Password to log into Airbyte | password |
port | Airbyte server port | 8000 |
use_ssl | If True, then service will use HTTPS | False |
username | Username to log into Airbyte | airbyte |
Example code
from mage_ai.services.airbyte import Airbyte
client = Airbyte(
api_version='v1',
host='host.docker.internal',
password='password',
port=8000,
use_ssl=False,
username='airbyte',
)
job = client.run_sync('7a749f2f-74b4-492e-9d13-30a3f390d111', poll_interval=2)
Sample result:
{
"connection_id": "7a749f2f-74b4-492e-9d13-30a3f390d111",
"connection_status": "active",
"job": {
"id": 9,
"configType": "sync",
"configId": "7a749f2f-74b4-492e-9d13-30a3f390d111",
"createdAt": 1671909838,
"updatedAt": 1671909843,
"status": "succeeded"
}
}
Example data loader
You can either explicitly hard code the connection_id in the data loader block
or you can add the value of the connection_id
as a runtime variable.
from mage_ai.services.airbyte import Airbyte
@data_loader
def load_data(*args, **kwargs):
connection_id = kwargs['connection_id']
client = Airbyte(
host='host.docker.internal',
password='password',
username='airbyte',
)
job = client.run_sync(connection_id, poll_interval=2)
return job