Implementing Layered Data Architecture In Dagster｜George T. C., Lai｜PyCon TW 2023
PyCon Taiwan 2023｜Talk 演講｜Day 1, R1 16:00–16:45
🪄 說明 Description 🪄
A layered data design pattern is a modern data architecture for building ETL/ELT data pipelines comprised of multiple stages so that each stage processes the data and improves the quality of the data progressively. Compared to the imperative way how data engineers build ETL/ELT data pipelines in the last decade, layered data architecture could be of great help in improving data quality steadily and progressively, and reducing data silos while project-specific teams are autonomously producing various data products. We will introduce, in this share, a technical solution based on layered data architecture. The solution is implemented by means of Dagster, a cloud-native data orchestrator with integrated lineage, observability, and a declarative programming model. A simple example will be presented in this talk to demonstrate concepts, principles, and data stack of the solution. In the end, the benefits we have gained from the implementation experience will be conveyed as well.
🪧 投影片 Slides：https://1drv.ms/p/s!AtNklwocKzYg8AEp6...
🚀 講者介紹 About Speaker - George T. C., Lai 🚀
A data practitioner with data analysis background who has been developing career mainly in Big Data and DevOps based on cloud-native ecosystem for 12 years. In the recent 7 years, I have been focusing on Data Architect, team management, and DevOps. As to technical experience, I got 6 years on Hadoop ecosystem, especially on Hortonworks HDP, 7 years on Kubernetes and 4 years on AWS/GCP. My personal vision is to make each data practitioner have a better life. I am approaching the vision by exploring new tools, discovering best practices, and delivering well-designed data architectures and technical solutions for data practitioners to relief their pain points and frustrations when coping with data.
Follow “PyCon Taiwan”
⭐️ Official Website: https://tw.pycon.org
⭐️ Facebook: / pycontw
⭐️ Instagram: / pycontw
⭐️ Twitter: / pycontw
⭐️ LinkedIn: / pycontw
⭐️ Blogger: https://conf.python.tw/
Watch video Implementing Layered Data Architecture In Dagster｜George T. C., Lai｜PyCon TW 2023 online without registration, duration hours minute second in high quality. This video was added by user PyCon Taiwan 22 November 2023, don't forget to share it with your friends and acquaintances, it has been viewed on our site 111 once and liked it 2 people.