This guide details adding a new source to Mage. If your source already exists as a Singer tap, check out our guide for adapting an existing source instead.
__init__.py
file that contains the logic for fetching data from the source.
test_connection
method that tests the connection to our source.discover
method that returns a Catalog of streams.load_data
method that yields data from the source as a dictionary./mage-ai/mage_integrations/mage_integrations/sources
and create a copy of the titanic
directory:
MY_SOURCE
directory:
config.json
file contains keys and values that are used to configure the behavior of the source as well as credentials to authenticate requests to the source. You must use the exact filename config.json
, regardless of your source’s name.
The following simple example is present in the Titanic source.
__init__.py
__init__.py
file. Most of the work in adding your own source will involve creating/overwriting methods from __init__.py
. Our sample contains the following:
load_data()
methodTitanic
source’s load_data
method reads data from a CSV file and yields a list of dictionaries:
load_data
method should also yield a list of dictionaries. There is a keyword argument named query
in the load_data
method that is a dictionary. When Mage runs a source, the following keys and values are automatically available on each run:
Key | Description | Sample value |
---|---|---|
_execution_date | The date and time (in ISO format) of when the pipeline started running. | 2022-10-21T17:24:49.443559 |
_execution_partition | An automatically formatted partition of the pipeline run using the execution date. | 20221021T172557 (e.g. format %Y%m%dT%H%M%S ) |
_start_date | You can define this variable as a runtime variable in your pipeline or it’ll be automatically filled in using the date and time your pipeline runs minus 1 hour, day, week, etc (based on your schedule’s interval). | 2022-10-01T00:00:00.000000 |
_end_date | You can define this variable as a runtime variable in your pipeline or it’ll be automatically filled in using the date and time your pipeline runs. | 2022-10-02T00:00:00.000000 |
discover()
methodtest_connection()
methodtest_connection
method to enable testing the tap in the Mage UI. This is a simple method that instantiates the tap and closes it. Just replace YourSource
with the source class, e.g. GitHub
and provide the configuration arguments to the tap. Here's a good example from the sFTP tap.
main()
functionmain()
function to match your new source:
README.md
file at the source’s root directory. Documentation helps Magers and future contributor understand exactly how your source works! Be sure to be thorough and descriptive. Here’s an example of documentation done well.