Building an End-to-End Data Observability System at Netflix with Joseph Machado
Building reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy.Joseph Machado, Senior Data Engineer at Netflix, joins us to share practical insights gleaned from supporting Netflix’s Ads business as well as over a decade of experience in the data engineering space. He discusses implementing audit publish patterns, building observability dashboards, defining in-band and separate data quality checks, and optimizing data validation across large-scale systems.Key Takeaways:.(03:14) Supporting data privacy and engineering efficiency within data systems.(10:41) Validating outputs with reconciliation checks to catch transformation issues.(16:06) Applying standardized patterns for auditing, validating and publishing data.(19:28) Capturing historical check results to monitor system health and improvements.(21:29) Treating data quality and availability as separate monitoring concerns.(26:26) Using containerization strategies to streamline pipeline executions.(29:47) Leveraging orchestration platforms for better visibility and retry capability.(31:59) Managing business pressure without sacrificing data quality practices.(35:46) Starting simple with quality checks and evolving toward more complex frameworks.Resources Mentioned:Joseph Machadohttps://www.linkedin.com/in/josephmachado1991/Netflix | LinkedInhttps://www.linkedin.com/company/netflix/Netflix | Websitehttps://www.netflix.com/browseStart Data Engineeringhttps://www.startdataengineering.com/Apache Airflowhttps://airflow.apache.org/dbt Labshttps://www.getdbt.com/Great Expectationshttps://greatexpectations.io/https://www.astronomer.io/events/roadshow/london/https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/https://www.astronomer.io/events/roadshow/san-francisco/https://www.astronomer.io/events/roadshow/chicago/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning