SFTP

​

Configuration

Connect securely to an SFTP server and load files into your Mage pipelines by providing the following configuration parameters:

​

Tables Configuration

Define how files are processed and structured into tables:

​

Required Parameters

​

Optional Parameters

​

Decryption Configuration (Optional)

If your files are encrypted with GPG, specify the following:

​

Mage Pro Enhancements 🚀

With Mage Pro, the SFTP source is enhanced for improved performance, usability, and scalability.
The following features are available in Mage Pro only:

• Auto-Generate Field Names:
  Automatically generate column names (col_0, col_1, …, col_N) when headers are missing by setting auto_generate_field_names: true.
• Schema Discovery Sample Rows:
  Speed up schema inference by limiting how many rows are sampled with schema_discovery_sample_rows.
• Improved Logging:
  Enhanced, structured logging for better traceability and debugging.
• TableSpec Schema Support:
  Define detailed table specifications for schema generation and validation.
• Selective Column Extraction:
  Write only the selected columns to the schema to reduce size and improve performance.
• Better Code Quality and Documentation:
  Cleaner, more robust code and fully updated documentation for enterprise use.

​

What is SFTP?

SFTP (Secure File Transfer Protocol) is a secure method for transferring files over an encrypted SSH connection.
It provides encrypted file access, transfer, and management across a reliable, secure channel.

​

Why Use SFTP with Mage?

• Secure and encrypted: Safely ingest sensitive files over SSH-secured channels.
• Seamless ETL workflows: Automate file ingestion into your data warehouse or lake.
• Support for encrypted files: Easily decrypt GPG-protected files during ingestion.
• High performance: Faster schema discovery, selective column loading, and large file handling.
• Enterprise-ready: Built for scale with advanced logging, security, and performance features.