Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero
Legacy architecture and AI workloads pose unique challenges at scale, especially in a global enterprise with complex data systems. In this episode, we explore strategies to proactively monitor and optimize pipelines while minimizing downstream failures.Adonis Castillo Cordero, Senior Automation Manager at Procter & Gamble, joins us to share actionable best practices for dependency mapping, anomaly detection and architecture simplification using Apache Airflow.Key Takeaways:(03:13) Integrating legacy data systems into modern architecture.(05:51) Designing workflows for real-time data processing.(07:57) Mapping dependencies early to avoid pipeline failures.(09:02) Building automated monitoring into orchestration frameworks.(12:09) Detecting anomalies to prevent performance bottlenecks.(15:24) Monitoring data quality to catch silent failures.(17:02) Prioritizing responses based on impact severity.(18:55) Simplifying dashboards to highlight critical metrics.Resources Mentioned:Adonis Castillo Corderohttps://www.linkedin.com/in/adoniscc/Procter & Gamble | LinkedInhttps://www.linkedin.com/company/procter-and-gamble/Procter & Gamble | Websitehttp://www.pg.comApache Airflowhttps://airflow.apache.org/OpenLineagehttps://openlineage.io/Azure Monitorhttps://azure.microsoft.com/en-us/products/monitor/AWS Lookout for Metricshttps://aws.amazon.com/lookout-for-metrics/Monte Carlohttps://www.montecarlodata.com/Great Expectationshttps://greatexpectations.io/https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning