Logging
Server Logging
The server logger can be configured to adjust the amount of detail that is included in the server logs. This can be helpful for debugging or troubleshooting server issues. You can change the verbosity level of the logger by setting the SERVER_VERBOSITY environment variable.
The SERVER_VERBOSITY variable accepts the following values: DEBUG, INFO, WARNING, ERROR and CRITICAL.
Pipelines logging
By default, logs for each pipeline run will be stored in the
<path_to_project>/pipelines/<pipeline_name>/.logs folder.
Learn more about logs here.
Set logging level
The default logging level is INFO. To customize the logging level for logs of block runs and pipeline runs,
you can set the level of logging_config in your project’s metadata.yaml file.
Example logging config to only log errors:
logging_config:
level: ERROR
Logging to external destination
S3
To store logs in S3, you need to set the logging config in your project’s metadata.yaml file.
Example S3 logging config:
logging_config:
type: s3
level: INFO
destination_config:
bucket: <bucket name>
prefix: <prefix path>
Google Cloud Storage
To store logs in GCS, you need to set the logging config in your project’s metadata.yaml file.
Example GCS logging config:
logging_config:
type: gcs
level: INFO
destination_config:
path_to_credentials: <path to gcp credentials json file>
bucket: <bucket name>
prefix: <prefix path>