Skip to main content

​
Prerequisites

Backblaze B2 uses Mage’s S3-compatible client, which depends on boto3. Docker and full dev installs already include it. If you installed Mage with a plain pip install mage-ai, install the s3 extra so the dependency is present: 
pip install "mage-ai[s3]"

​
Add credentials

  1. Create a new pipeline or open an existing pipeline.
  2. Expand the left side of your screen to view the file browser.
  3. Scroll down and click on a file named io_config.yaml.
  4. Enter the following keys and values under the key named default (you can have multiple profiles, add it under whichever is relevant to you) 
    version: 0.1.1
default:
  B2_APPLICATION_KEY_ID: ...
  B2_APPLICATION_KEY: ...
  # Optional. Defaults to us-west-004. If your bucket is in another region,
  # uncomment and set this to that region's endpoint (shown on the B2 bucket
  # page), e.g. us-west-001, us-east-005, eu-central-003:
  # B2_ENDPOINT_URL: https://s3.us-east-005.backblazeb2.com
    For backwards compatibility, the loader also falls back to AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_ENDPOINT (for the endpoint URL) when the corresponding B2_* keys are not set.

​
Using Python block

  1. Create a new pipeline or open an existing pipeline.
  2. Add a data loader or transformer block (the code snippet below is for a data loader).
  3. Select Generic (no template).
  4. Enter this code snippet (note: change the config_profile from default if you have a different profile): 
    from mage_ai.settings.repo import get_repo_path
from mage_ai.io.config import ConfigFileLoader
from mage_ai.io.backblaze_b2 import BackblazeB2
from os import path
from pandas import DataFrame

if 'data_loader' not in globals():
    from mage_ai.data_preparation.decorators import data_loader


@data_loader
def load_from_backblaze_b2(**kwargs) -> DataFrame:
    config_path = path.join(get_repo_path(), 'io_config.yaml')
    config_profile = 'default'

    bucket_name = '...'  # Change to your B2 bucket name
    object_key = '...'   # Change to your object key

    return BackblazeB2.with_config(ConfigFileLoader(config_path, config_profile)).load(
        bucket_name,
        object_key,
    )
  5. Run the block.

​
Endpoint override

The default endpoint is https://s3.us-west-004.backblazeb2.com. If your bucket lives in another B2 region, add: 
default:
  B2_ENDPOINT_URL: https://s3.us-east-005.backblazeb2.com
under your profile in the io_config.yaml file. AWS_ENDPOINT is honored as a fallback.

​
Errors

B2 connection endpoint URL error Open the io_config.yaml file at the root of your project (e.g. default_repo/io_config.yaml) and confirm that B2_ENDPOINT_URL (if set) matches the region of your B2 bucket.

​
Permissions

Ensure the application key you create in the Backblaze B2 console grants the following capabilities on your bucket:
  • readFiles
  • writeFiles
  • listFiles
These permissions are required to:
  • Read data from B2 (e.g. .csv, .parquet, .json)
  • Write query results or transformed data to B2
  • List contents of a bucket when needed
Two common Backblaze B2 gotchas:
  • The master application key is not S3-compatible. Create a standard (non-master) application key for use with Mage.
  • App keys restricted to a single bucket may also need the listAllBucketNames capability for S3 SDK/integration compatibility.