Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.mage.ai/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

HashiCorp Vault is a secrets management tool that provides secure storage and access to sensitive data. Mage integrates with Vault to retrieve secrets at runtime, ensuring that sensitive credentials are never stored in your codebase or configuration files.

​
Prerequisites

Before using Vault with Mage, ensure you have:
  1. A running HashiCorp Vault instance
  2. Appropriate permissions to read secrets from Vault
  3. A valid Vault token or authentication method

​
Configuration

​
Required Environment Variables

Configure your Mage environment with the following environment variables:
  • VAULT_URL: The URL of your Vault server (default: http://127.0.0.1:8200)
  • VAULT_TOKEN: Your Vault authentication token

​
Optional Environment Variables

  • VAULT_MOUNT_POINT: The KV v2 mount point for secrets (default: secret)

​
Example Environment Setup

export VAULT_URL="https://vault.company.com"
export VAULT_TOKEN="hvs.xxxxxxxxxxxxxxxxxxxx"
export VAULT_MOUNT_POINT="secret"

​
Using Secrets in Mage

​
Python Code

Use the Vault integration directly in your Python blocks. The get_secret() function accepts two parameters:
  • path: The secret path in Vault (e.g., 'mage/database')
  • key: Optional specific key within the secret (e.g., 'password')
Syntax: 
from mage_ai.services.hashicorp.vault.vault import get_secret

# Get a specific key from a secret
value = get_secret('secret/path', 'key_name')

# Get all key-value pairs from a secret
all_data = get_secret('secret/path')
Example: 
from mage_ai.services.hashicorp.vault.vault import get_secret

# Get a specific key from a secret
db_password = get_secret('mage/database', 'password')

# Get all key-value pairs from a secret
db_config = get_secret('mage/database')
print(f"Database host: {db_config['host']}")
print(f"Database username: {db_config['username']}")

​
YAML Configuration

Reference Vault secrets in your YAML configuration files using the vault_secret_var() function. This function accepts two parameters:
  • path: The secret path in Vault (e.g., 'mage/database')
  • key: The specific key within the secret (e.g., 'password')
Syntax: Use the "{{ vault_secret_var('secret/path', 'key_name') }}" syntax in any YAML field that supports variable interpolation: 
version: 0.1.1
default:
  VARIABLE_NAME: "{{ vault_secret_var('secret/path', 'key_name') }}"
Example 1 - IO Config: 
version: 0.1.1
default:
  # PostgreSQL configuration using Vault secrets
  POSTGRES_HOST: "{{ vault_secret_var('mage/database', 'host') }}"
  POSTGRES_PORT: 5432
  POSTGRES_DBNAME: "my_database"
  POSTGRES_USER: "{{ vault_secret_var('mage/database', 'username') }}"
  POSTGRES_PASSWORD: "{{ vault_secret_var('mage/database', 'password') }}"
Example 2 - Data Integration Pipeline: 
# API source configuration
url: "https://api.example.com/data"
method: GET
headers:
  Authorization: "Bearer {{ vault_secret_var('mage/api', 'token') }}"
  X-API-Key: "{{ vault_secret_var('mage/api', 'key') }}"
response_parser: data.items

​
Creating Secrets in Vault

​
Using Vault CLI

  1. Authenticate with Vault: 
    vault auth -method=userpass username=your-username
  2. Create a secret: 
    vault kv put secret/mage/database \
  username=dbuser \
  password=dbpassword \
  host=db.example.com
  3. Verify the secret: 
    vault kv get secret/mage/database

​
Using Vault UI

  1. Navigate to your Vault UI:
    • Local development: http://127.0.0.1:8200/ui
    • Production: https://vault.company.com/ui
  2. Go to Secrets → KV v2
  3. Click Create secret
  4. Enter the secret path (e.g., mage/database)
  5. Add key-value pairs for your credentials
  6. Click Save

​
Error Handling

The Vault integration provides specific error handling for common scenarios: 
from mage_ai.services.hashicorp.vault.vault import get_secret

try:
    secret_value = get_secret('mage/database', 'password')
except KeyError as e:
    print(f"Secret or key not found: {e}")
except PermissionError as e:
    print(f"Access denied: {e}")
except Exception as e:
    print(f"Vault error: {e}")

​
Security Best Practices

​
1. Token Management

  • Use short-lived tokens when possible
  • Implement token rotation
  • Store tokens securely (environment variables, not in code)

​
2. Secret Organization

  • Use hierarchical paths (e.g., mage/prod/database, mage/dev/database)
  • Follow consistent naming conventions
  • Separate secrets by environment

​
3. Access Control

  • Implement least-privilege access
  • Use Vault policies to restrict access
  • Regularly audit secret access

​
4. Monitoring

  • Enable Vault audit logs
  • Monitor secret access patterns
  • Set up alerts for unusual activity

​
Troubleshooting

​
Common Issues

1. Authentication Failed 
Exception: Unable to authenticate to Vault at https://vault.company.com
  • Verify your VAULT_TOKEN is valid
  • Check if the token has expired
  • Ensure the Vault URL is correct
2. Secret Not Found 
KeyError: Secret path 'mage/database' not found in Vault
  • Verify the secret path exists
  • Check the mount point configuration
  • Ensure you have read permissions
3. Access Denied 
PermissionError: Access denied to secret path 'mage/database'
  • Check your Vault token permissions
  • Verify the secret path is accessible
  • Contact your Vault administrator

​
Advanced Usage

​
Custom Mount Points

If your Vault instance uses a different mount point: 
export VAULT_MOUNT_POINT="custom-secrets"

​
Multiple Vault Instances

For different environments, you can create multiple Vault clients: 
import os
from mage_ai.services.hashicorp.vault.vault import Vault

# Override environment variables for specific operations
os.environ['VAULT_URL'] = 'https://vault-prod.company.com'
os.environ['VAULT_TOKEN'] = 'prod-token'

prod_vault = Vault()

​
Integration Examples

​
Database Connection

from mage_ai.services.hashicorp.vault.vault import get_secret
import psycopg2

# Get database credentials from Vault
db_config = get_secret('mage/database')

# Create database connection
conn = psycopg2.connect(
    host=db_config['host'],
    port=db_config['port'],
    database=db_config['database'],
    user=db_config['username'],
    password=db_config['password']
)

​
API Authentication

from mage_ai.services.hashicorp.vault.vault import get_secret
import requests

# Get API credentials from Vault
api_token = get_secret('mage/api', 'token')

# Make authenticated API request
headers = {'Authorization': f'Bearer {api_token}'}
response = requests.get('https://api.example.com/data', headers=headers)

​
Cloud Provider Credentials

from mage_ai.services.hashicorp.vault.vault import get_secret
import boto3

# Get AWS credentials from Vault
aws_creds = get_secret('mage/aws')

# Create AWS client
session = boto3.Session(
    aws_access_key_id=aws_creds['access_key_id'],
    aws_secret_access_key=aws_creds['secret_access_key'],
    region_name=aws_creds['region']
)

s3_client = session.client('s3')

​
Migration from Other Secret Managers

If you’re migrating from other secret management solutions:

​
From Mage Built-in Secrets

# Old way
from mage_ai.data_preparation.shared.secrets import get_secret_value
password = get_secret_value('database_password')

# New way
from mage_ai.services.hashicorp.vault.vault import get_secret
password = get_secret('mage/database', 'password')

​
From Environment Variables

# Old way
import os
password = os.getenv('DATABASE_PASSWORD')

# New way
from mage_ai.services.hashicorp.vault.vault import get_secret
password = get_secret('mage/database', 'password')

​
Support

For additional help with Vault integration: