This guide details how to monitor your
mage_ai application using OpenTelemetry. OpenTelemetry provides a unified approach to collecting telemetry data, such as traces and metrics, which are essential for understanding your application’s performance and behavior.
Ensure the following prerequisites are in place before proceeding:
- An OpenTelemetry Collector is set up and configured to receive telemetry data.
mage_aiapplication is ready for instrumentation with OpenTelemetry.
Configuring OpenTelemetry in
Setting Environment Variables
Configure the OpenTelemetry Exporters in your application by setting the following environment variables:
OTLP HTTP Exporter: Use
OTEL_EXPORTER_OTLP_HTTP_ENDPOINTto specify the HTTP endpoint of your OpenTelemetry Collector.
The OTEL_EXPORTER_OTLP_ENDPOINT environment variable sets a general collector endpoint, which can be used for both gRPC and HTTP connections.
Instrumentation in the Application
SQLAlchemy Instrumentation: The application uses the SQLAlchemyInstrumentor from OpenTelemetry to instrument database operations. This generates telemetry data like traces for database interactions.
Tornado Instrumentation: To monitor HTTP server operations, the application integrates OpenTelemetry’s Tornado instrumentation. This collects valuable data related to HTTP requests and server performance.
Telemetry Data Collection and Analysis
With the environment variables set and the application running, OpenTelemetry will start collecting telemetry data based on the defined instrumentation.
Data Collection: The application will send the collected telemetry data, including traces and metrics, to the specified OpenTelemetry Collector endpoint.
Analyzing Telemetry Data: For effective visualization and analysis, you can connect your OpenTelemetry Collector to backend tools that support OpenTelemetry data (like Grafana or Jaeger).