Workflow Organization for Cloud Composer
Contact us to book this course
Learning Track
Data Engineering and Analytics
Delivery methods
On-Site, Virtual
Duration
1 day
Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. Composer enables you to create, schedule, monitor, and manage workflow pipelines that span across clouds and on-premises data centers. In this course, you will learn about Apache Airflow and its implementation via Cloud Composer. You will learn how to provision Composer instances, create and manage Airflow DAGs on Composer, and perform tasks such as testing, debugging, and monitoring of Airflow DAGs.
Learning objectives
- Explore Apache Airflow and Cloud Composer as workflow orchestration solutions.
- Create and manage Airflow DAGs following best practices.
- Test and debug Airflow DAGs.
- Monitor and observe Airflow DAGs on Cloud Composer.
Course outline
- Data Engineer's need for Workflow Orchestration
- Introduction to Apache Airflow
- Cloud Composer
- Environment Setup
- Using the Composer and Airflow
- Lab: Provisioning Cloud Composer
- DAG structure and best practices
- Common operators • Dependencies, trigger rules, and flow control
- Integration of Airflow and Google Cloud Services
- Lab: Assembling a Data Processing Workflow
- Advanced Airflow features
- Debugging DAGs
- Performance and scalability
- Security and Access Control
- Observability and monitoring
- Lab: Extending and Monitoring DAGs
