​
Airbyte cloud

​
Configuration

Mage Pro supports triggering Airbyte Cloud connection syncs directly from your data pipelines. To connect to Airbyte Cloud, you need to configure the following keyword arguments.

Keyword argumentDescriptionDefault value
use_cloudWheter to use Airbyte CloudFalse
client_idAirbyte cloud application client_idNone
client_secretAirbyte cloud application client_secretNone

Refer to the Airbyte documentation to retrieve your client_id and client_secret.

​
Example data loader

You can either explicitly hard code the connection_id in the data loader block or you can add the value of the connection_id as a runtime variable.

from mage_ai.services.airbyte import Airbyte

from mage_ai.services.airbyte import Airbyte


@data_loader
def load_data(*args, **kwargs):
    connection_id = kwargs['connection_id']
    client = Airbyte(
        use_cloud=True,
        client_id='client_id',
        client_secret='client_secret',
    )
    job = client.run_sync(connection_id, poll_interval=2)

    return job

​
Open source

​
Configuration

Here are the following keyword arguments that can be used to configure Airbyte:

Keyword argumentDescriptionDefault value
api_versionAPI versionv1
hostAirbyte server hosthost.docker.internal
passwordPassword to log into Airbytepassword
portAirbyte server port8000
use_sslIf True, then service will use HTTPSFalse
usernameUsername to log into Airbyteairbyte

​
Example code

from mage_ai.services.airbyte import Airbyte


client = Airbyte(
    api_version='v1',
    host='host.docker.internal',
    password='password',
    port=8000,
    use_ssl=False,
    username='airbyte',
)
job = client.run_sync('7a749f2f-74b4-492e-9d13-30a3f390d111', poll_interval=2)

Sample result:

{
  "connection_id": "7a749f2f-74b4-492e-9d13-30a3f390d111",
  "connection_status": "active",
  "job": {
    "id": 9,
    "configType": "sync",
    "configId": "7a749f2f-74b4-492e-9d13-30a3f390d111",
    "createdAt": 1671909838,
    "updatedAt": 1671909843,
    "status": "succeeded"
  }
}

​
Example data loader

You can either explicitly hard code the connection_id in the data loader block or you can add the value of the connection_id as a runtime variable.

from mage_ai.services.airbyte import Airbyte


@data_loader
def load_data(*args, **kwargs):
    connection_id = kwargs['connection_id']
    client = Airbyte(
        host='host.docker.internal',
        password='password',
        username='airbyte',
    )
    job = client.run_sync(connection_id, poll_interval=2)

    return job

Was this page helpful?

Suggest editsRaise issue
PrefectStitch