Adding an IO class
IO classes power Mage sources and destinations. Read on to learn how you can contribute an IO class to Mage.
What’s an IO class?
IO classes are at the heart of Mage— they’re the code that enables Mage to read and write data. When you configure your io_config
to use a source or destination, you’re telling Mage to use a specific IO class.
IO classes are defined here in the Mage repository. You can see that there are a few different types of IO classes— some are for reading data, some are for writing data, and some are for both.
Why should I contribute?
If you have a favorite IO class that’s not currently supported, you can reach out or contribute it to Mage! Contributing is a great way to build your skills, become a part of the Mage community, and help others.
How do I contribute?
Configure your development environment
The first step to contributing an IO class is to configure your development environment. You can find instructions for doing so here.
Create a new IO class
Once configured, you’ll want to create a new file in the mage_ai/io
directory. The file should be named after the IO class you’re contributing. For example, if you’re contributing an IO class called MyIOClass
, you should create a file called my_io_class.py
.
Some IO classes are relatively straightforward, like exporting to local files or writing to Google Sheets. Others, like full database integrations, can be pretty complex. Google BigQuery is a good example of a complex IO class.
Most classes are platform specific, but for a database, you might need the following methods:
alter_table
: Alter the table schemaload
: Load data from the databaseexport
: Export data to the databaseexecute
: Execute a query on the databaseexecute_queries
: Execute multiple queries on the database
Additionally, every class will need to have the following methods:
__init__
: Initialize the classwith_config
: Initialize the database client from the configuration loader. You’ll notice that this method is used in all of our templates.
Test your IO class
In order to test your class, you should perform all of the methods you defined in the class itself. Once you pass our linting checks (outlined in the development environment link above) and your tests pass, you’re ready to submit a pull request! Learn more about pull requests on the contribution.
Create a pull request
Fork the Mage GitHub repo
Visit GitHub and fork the Mage repo, then clone it to your machine.
Create a branch
Create a branch in your forked repo and commit your changes.
Create a pull request
Once you've made and tested your changes, create a pull request on the Mage repo.
Tag Mage team members for a review
Add members for review in GitHub, please tag one of the following:
@wangxiaoyou1993
(backend/IO)@dy46
(backend)@johnson-mage
(frontend/docs/website)@tommydangerous
Was this page helpful?