Airbyte cloud

Only in Mage Pro.Try our fully managed solution to access this advanced feature.

Configuration

Mage Pro supports triggering Airbyte Cloud connection syncs directly from your data pipelines. To connect to Airbyte Cloud, you need to configure the following keyword arguments.

Keyword argument	Description	Default value
`use_cloud`	Whether to use Airbyte Cloud	`False`
`client_id`	Airbyte cloud application client_id	`None`
`client_secret`	Airbyte cloud application client_secret	`None`

Refer to the Airbyte documentation to retrieve your client_id and client_secret.

Example data loader

You can either explicitly hard code the connection_id in the data loader block or you can add the value of the connection_id as a runtime variable. from mage_ai.services.airbyte import Airbyte

from mage_ai.services.airbyte import Airbyte


@data_loader
def load_data(*args, **kwargs):
    connection_id = kwargs['connection_id']
    client = Airbyte(
        use_cloud=True,
        client_id='client_id',
        client_secret='client_secret',
    )
    job = client.run_sync(connection_id, poll_interval=2)

    return job

Open source

Configuration

Here are the following keyword arguments that can be used to configure Airbyte:

Keyword argument	Description	Default value
`api_version`	API version	`v1`
`host`	Airbyte server host	`host.docker.internal`
`password`	Password to log into Airbyte	`password`
`port`	Airbyte server port	`8000`
`use_ssl`	If `True`, then service will use HTTPS	`False`
`username`	Username to log into Airbyte	`airbyte`

Example code

from mage_ai.services.airbyte import Airbyte


client = Airbyte(
    api_version='v1',
    host='host.docker.internal',
    password='password',
    port=8000,
    use_ssl=False,
    username='airbyte',
)
job = client.run_sync('7a749f2f-74b4-492e-9d13-30a3f390d111', poll_interval=2)

Sample result:

{
  "connection_id": "7a749f2f-74b4-492e-9d13-30a3f390d111",
  "connection_status": "active",
  "job": {
    "id": 9,
    "configType": "sync",
    "configId": "7a749f2f-74b4-492e-9d13-30a3f390d111",
    "createdAt": 1671909838,
    "updatedAt": 1671909843,
    "status": "succeeded"
  }
}

Example data loader

You can either explicitly hard code the connection_id in the data loader block or you can add the value of the connection_id as a runtime variable.

from mage_ai.services.airbyte import Airbyte


@data_loader
def load_data(*args, **kwargs):
    connection_id = kwargs['connection_id']
    client = Airbyte(
        host='host.docker.internal',
        password='password',
        username='airbyte',
    )
    job = client.run_sync(connection_id, poll_interval=2)

    return job

Infrastructure

3rd party

Airbyte in Mage

Airbyte cloud

Configuration

Example data loader

Open source

Configuration

Example code

Example data loader

​Airbyte cloud

​Configuration

​Example data loader

​Open source

​Configuration

​Example code

​Example data loader

Airbyte cloud

Configuration

Example data loader

Open source

Configuration

Example code

Example data loader