HashiCorp Vault

Only in Mage Pro.Try our fully managed solution to access this advanced feature.

Overview

HashiCorp Vault is a secrets management tool that provides secure storage and access to sensitive data. Mage integrates with Vault to retrieve secrets at runtime, ensuring that sensitive credentials are never stored in your codebase or configuration files.

Prerequisites

Before using Vault with Mage, ensure you have:

A running HashiCorp Vault instance
Appropriate permissions to read secrets from Vault
A valid Vault token or authentication method

Configuration

Required Environment Variables

Configure your Mage environment with the following environment variables:

VAULT_URL: The URL of your Vault server (default: http://127.0.0.1:8200)
VAULT_TOKEN: Your Vault authentication token

Optional Environment Variables

VAULT_MOUNT_POINT: The KV v2 mount point for secrets (default: secret)

Example Environment Setup

export VAULT_URL="https://vault.company.com"
export VAULT_TOKEN="hvs.xxxxxxxxxxxxxxxxxxxx"
export VAULT_MOUNT_POINT="secret"

Using Secrets in Mage

Python Code

Use the Vault integration directly in your Python blocks. The get_secret() function accepts two parameters:

path: The secret path in Vault (e.g., 'mage/database')
key: Optional specific key within the secret (e.g., 'password')

Syntax:

from mage_ai.services.hashicorp.vault.vault import get_secret

# Get a specific key from a secret
value = get_secret('secret/path', 'key_name')

# Get all key-value pairs from a secret
all_data = get_secret('secret/path')

Example:

from mage_ai.services.hashicorp.vault.vault import get_secret

# Get a specific key from a secret
db_password = get_secret('mage/database', 'password')

# Get all key-value pairs from a secret
db_config = get_secret('mage/database')
print(f"Database host: {db_config['host']}")
print(f"Database username: {db_config['username']}")

YAML Configuration

Reference Vault secrets in your YAML configuration files using the vault_secret_var() function. This function accepts two parameters:

path: The secret path in Vault (e.g., 'mage/database')
key: The specific key within the secret (e.g., 'password')

Syntax: Use the "{{ vault_secret_var('secret/path', 'key_name') }}" syntax in any YAML field that supports variable interpolation:

version: 0.1.1
default:
  VARIABLE_NAME: "{{ vault_secret_var('secret/path', 'key_name') }}"

Example 1 - IO Config:

version: 0.1.1
default:
  # PostgreSQL configuration using Vault secrets
  POSTGRES_HOST: "{{ vault_secret_var('mage/database', 'host') }}"
  POSTGRES_PORT: 5432
  POSTGRES_DBNAME: "my_database"
  POSTGRES_USER: "{{ vault_secret_var('mage/database', 'username') }}"
  POSTGRES_PASSWORD: "{{ vault_secret_var('mage/database', 'password') }}"

Example 2 - Data Integration Pipeline:

# API source configuration
url: "https://api.example.com/data"
method: GET
headers:
  Authorization: "Bearer {{ vault_secret_var('mage/api', 'token') }}"
  X-API-Key: "{{ vault_secret_var('mage/api', 'key') }}"
response_parser: data.items

Creating Secrets in Vault

Using Vault CLI

Authenticate with Vault:

vault auth -method=userpass username=your-username

Create a secret:

vault kv put secret/mage/database \
  username=dbuser \
  password=dbpassword \
  host=db.example.com

Verify the secret:
```
vault kv get secret/mage/database
```

Using Vault UI

Navigate to your Vault UI:
- Local development: http://127.0.0.1:8200/ui
- Production: https://vault.company.com/ui
Go to Secrets → KV v2
Click Create secret
Enter the secret path (e.g., mage/database)
Add key-value pairs for your credentials
Click Save

Error Handling

The Vault integration provides specific error handling for common scenarios:

from mage_ai.services.hashicorp.vault.vault import get_secret

try:
    secret_value = get_secret('mage/database', 'password')
except KeyError as e:
    print(f"Secret or key not found: {e}")
except PermissionError as e:
    print(f"Access denied: {e}")
except Exception as e:
    print(f"Vault error: {e}")

Security Best Practices

1. Token Management

Use short-lived tokens when possible
Implement token rotation
Store tokens securely (environment variables, not in code)

2. Secret Organization

Use hierarchical paths (e.g., mage/prod/database, mage/dev/database)
Follow consistent naming conventions
Separate secrets by environment

3. Access Control

Implement least-privilege access
Use Vault policies to restrict access
Regularly audit secret access

4. Monitoring

Enable Vault audit logs
Monitor secret access patterns
Set up alerts for unusual activity

Troubleshooting

Common Issues

1. Authentication Failed

Exception: Unable to authenticate to Vault at https://vault.company.com

Verify your VAULT_TOKEN is valid
Check if the token has expired
Ensure the Vault URL is correct

2. Secret Not Found

KeyError: Secret path 'mage/database' not found in Vault

Verify the secret path exists
Check the mount point configuration
Ensure you have read permissions

3. Access Denied

PermissionError: Access denied to secret path 'mage/database'

Check your Vault token permissions
Verify the secret path is accessible
Contact your Vault administrator

Advanced Usage

Custom Mount Points

If your Vault instance uses a different mount point:

export VAULT_MOUNT_POINT="custom-secrets"

Multiple Vault Instances

For different environments, you can create multiple Vault clients:

import os
from mage_ai.services.hashicorp.vault.vault import Vault

# Override environment variables for specific operations
os.environ['VAULT_URL'] = 'https://vault-prod.company.com'
os.environ['VAULT_TOKEN'] = 'prod-token'

prod_vault = Vault()

Integration Examples

Database Connection

from mage_ai.services.hashicorp.vault.vault import get_secret
import psycopg2

# Get database credentials from Vault
db_config = get_secret('mage/database')

# Create database connection
conn = psycopg2.connect(
    host=db_config['host'],
    port=db_config['port'],
    database=db_config['database'],
    user=db_config['username'],
    password=db_config['password']
)

API Authentication

from mage_ai.services.hashicorp.vault.vault import get_secret
import requests

# Get API credentials from Vault
api_token = get_secret('mage/api', 'token')

# Make authenticated API request
headers = {'Authorization': f'Bearer {api_token}'}
response = requests.get('https://api.example.com/data', headers=headers)

Cloud Provider Credentials

from mage_ai.services.hashicorp.vault.vault import get_secret
import boto3

# Get AWS credentials from Vault
aws_creds = get_secret('mage/aws')

# Create AWS client
session = boto3.Session(
    aws_access_key_id=aws_creds['access_key_id'],
    aws_secret_access_key=aws_creds['secret_access_key'],
    region_name=aws_creds['region']
)

s3_client = session.client('s3')

Migration from Other Secret Managers

If you’re migrating from other secret management solutions:

From Mage Built-in Secrets

# Old way
from mage_ai.data_preparation.shared.secrets import get_secret_value
password = get_secret_value('database_password')

# New way
from mage_ai.services.hashicorp.vault.vault import get_secret
password = get_secret('mage/database', 'password')

From Environment Variables

# Old way
import os
password = os.getenv('DATABASE_PASSWORD')

# New way
from mage_ai.services.hashicorp.vault.vault import get_secret
password = get_secret('mage/database', 'password')

Support

For additional help with Vault integration:

Deployment

Cloud infrastructure

Compute resources

Secret manager

Authentication

HashiCorp Vault

Overview

Prerequisites

Configuration

Required Environment Variables

Optional Environment Variables

Example Environment Setup

Using Secrets in Mage

Python Code

YAML Configuration

Creating Secrets in Vault

Using Vault CLI

Using Vault UI

Error Handling

Security Best Practices

1. Token Management

2. Secret Organization

3. Access Control

4. Monitoring

Troubleshooting

Common Issues

Advanced Usage

Custom Mount Points

Multiple Vault Instances

Integration Examples

Database Connection

API Authentication

Cloud Provider Credentials

Migration from Other Secret Managers

From Mage Built-in Secrets

From Environment Variables

Support

Deployment

Cloud infrastructure

Compute resources

Secret manager

Authentication

​Overview

​Prerequisites

​Configuration

​Required Environment Variables

​Optional Environment Variables

​Example Environment Setup

​Using Secrets in Mage

​Python Code

​YAML Configuration

​Creating Secrets in Vault

​Using Vault CLI

​Using Vault UI

​Error Handling

​Security Best Practices

​1. Token Management

​2. Secret Organization

​3. Access Control

​4. Monitoring

​Troubleshooting

​Common Issues

​Advanced Usage

​Custom Mount Points

​Multiple Vault Instances

​Integration Examples

​Database Connection

​API Authentication

​Cloud Provider Credentials

​Migration from Other Secret Managers

​From Mage Built-in Secrets

​From Environment Variables

​Support

Overview

Prerequisites

Configuration

Required Environment Variables

Optional Environment Variables

Example Environment Setup

Using Secrets in Mage

Python Code

YAML Configuration

Creating Secrets in Vault

Using Vault CLI

Using Vault UI

Error Handling

Security Best Practices

1. Token Management

2. Secret Organization

3. Access Control

4. Monitoring

Troubleshooting

Common Issues

Advanced Usage

Custom Mount Points

Multiple Vault Instances

Integration Examples

Database Connection

API Authentication

Cloud Provider Credentials

Migration from Other Secret Managers

From Mage Built-in Secrets

From Environment Variables

Support