Mage builds data integrations from Singer taps, so if there’s a Singer source you want to use that isn’t supported yet, you can adapt it yourself!
/mage_integrations/mage_integrations
folder. To add a new source, you’ll need to create a new folder in the /sources
subdirectory.
The folder name should be the name of the tap. For existing sources, that starts with pulling in the GitHub repo for the tap. For example, for the GitHub tap, we would:
tap_your_source
and tests
, but hold onto anything that looks like a config.json
file or a sample config.
Next, we need to create a configuration file and an init file:
config.json
file with the sample from the tap. In the GitHub example, this is config.sample.json
. It looks like this:
your_tap/__init__.py
:
Github
for Github, and tap_your_tap
with the name of your tap, e.g. tap_github
.
discover
and sync
methods to make them work with Mage. Discover is likely in tap_your_tap/discover.py
and sync is likely in tap_your_tap/sync.py
. It requires a bit of nuance, but our job is to look through the existing Singer tap, understand how the sync
and discover
methods were implemented, then execute them in Mage __init__.py
under the corresponding method. This is necessarily different for every tap, since each project is entirely independent.
Don’t forget to use absolute imports for these files. HubSpot is another good example of a completed tap.
test_connection
method to enable testing the tap in the Mage UI. This is a simple method that instantiates the tap and closes it. Just replace YourSource
with the source class, e.g. GitHub
and provide the configuration arguments to the tap. Here's a good example from the sFTP tap.
write_schema
methodwrite_schema
method in the tap_your_tap/sync.py
file. Add the following import:
from mage_integrations.sources.messages import write_schema
Now, switch out the default function (likely singer.write_schema
) for the mage function using the default arguments. Here’s an example:
tap_github/sync.py
, tap_github/discover.py
:
LOGGER
with mage_logger
.
SOURCES
list constant in this file
This will make your new source visible in the Mage UI.
README.md
/sources/github/README.md
. This should include: