Skip to main content
Pipeline automation is a critical component of production data workflows It ensures consistent and reliable execution without manual intervention. Mage Pro provides comprehensive orchestration capabilities through its trigger UI or YAML conde configuration, which initiates pipeline execution based on predefined conditions or external events. Mage supports two primary orchestration patterns for pipeline automation: 1. Schedule-Based Orchestration Schedule triggers provide time-based automation capabilities with flexible configuration options:
  • Continuous Execution: Maintains persistent pipeline runs for real-time data processing scenarios
  • Interval-Based Execution: Configures pipelines to run at specified intervals (hourly, daily, weekly, etc.)
  • One-Time Execution: Schedules pipelines for single execution at a designated time
  • Custom Timeline Configuration: Enables complex scheduling patterns with custom cron expressions
  • Deadline Management: Sets completion time constraints to ensure pipelines finish within specified windows
This method is optimal for batch processing workflows, ETL operations, and scenarios requiring predictable execution patterns. 2. API-Driven Orchestration API triggers enable event-driven pipeline execution through HTTP endpoints:
  • RESTful Integration: Exposes pipeline execution through standardized POST requests
  • External System Integration: Allows third-party applications to trigger pipeline runs
  • Programmatic Control: Enables dynamic pipeline orchestration through application logic
This approach is ideal for real-time processing scenarios, webhook integrations, and microservice architectures requiring on-demand pipeline execution. Let’s create a new schedule based trigger: Step 1: From the pipeline editor, non-viewer users can select Run pipeline in the header or Run menu to open the Triggers page. Viewer users and users in compact layouts should use the existing left popout navigation menu and select Triggers. Step 2: Once on the triggers page click the green “New trigger” button Create new pipeline Step 3: After clicking “New Trigger,” you will be taken to a screen where you can create a trigger. Select the “Schedule” option and the UI pictured below ill populate. Step 4: Complete the text fields below to configure your trigger based on the workflows requirements. You can also configure Run settings, SLA requirements, and Runtime variables in this UI. Step 5: Click Create trigger to save your trigger. Create new pipeline Step 6: After saving your changes you will be taken to a new screen where you can enable your trigger. Once enabled, the trigger will begin to run based on your previous configuration. Step 7: Click the “Save trigger in code” button to save your trigger to a YAML file. Create new pipeline Once your trigger is enabled, Frequency is the trigger plan: it controls the run schedule and actual run time. Schedule start only controls when that plan is allowed to begin. Choose Start at next Frequency time to let the trigger run at the next scheduled Frequency time, or choose Do not start before date/time to set the earliest allowed start. That date/time is not the run time; Frequency still controls when runs happen. You have successfully automated your first Mage pipeline. This foundational automation capability enables consistent, reliable data processing without manual intervention. You can now apply these same principles to build additional automated workflows for your data infrastructure.