Overview

Use OpenSearch as a destination in Mage to publish JSON-formatted records into an OpenSearch index. This is ideal for search indexing, log aggregation, and real-time analytics.

By default, Mage uses the pipeline’s table name to set the OpenSearch index name. You can also define dynamic index patterns using JSONPath expressions.


Configuration Parameters

KeyDescriptionDefaultRequired
schemeHTTP scheme used for connecting to OpenSearch. Use http or https.http
hostHostname or IP address of the OpenSearch node or cluster.localhost
portPort number for the OpenSearch HTTP endpoint.9200
username(Optional) Username for basic authentication.None
password(Optional) Password for basic authentication.None
bearer_token(Optional) Bearer token for token-based authentication.None
api_key_id(Optional) API key ID for key-based authorization.None
api_key(Optional) API key secret for key-based authorization.None
ssl_ca_file(Optional) Path to the SSL CA certificate file for verifying HTTPS connections.None
index_schema_fields(Optional) JSONPath mapping used to dynamically generate the index name based on record values.None
metadata_fields(Optional) Dictionary mapping fields (using JSONPath) to populate ECS-style metadata fields in the index request (e.g., _id, @timestamp).None

Notes

  • Mage automatically converts each record into JSON format and indexes it into the specified OpenSearch index.
  • If index_schema_fields is configured, Mage will use the JSONPath-extracted values from each record to generate dynamic index names.
  • metadata_fields is useful for setting OpenSearch ECS-compatible fields like _id, @timestamp, etc., from the incoming data stream.
  • Authentication can be configured using basic auth, bearer tokens, or API key pairs.
  • For production use, we recommend enabling https and providing a trusted ssl_ca_file.