ETL Incremental Data Load Approach Using DLT | Source Change Detection | Load New & Change Data
In this video we use the data load tool (dlt) library. We will explore the date based incremental data load using dlt. Previously we have covered the auto incrementing id incremental load approach. We use the source change detection technique. With this approach we detect what rows have changed and only pull the change data for the ETL operation. This is an optimal method of moving between systems.
Previous video: • How to perform ETL Incremental Data L...
Link to GitHub repo: https://github.com/hnawaz007/pythonda...
DLT docs on incremental load: https://dlthub.com/docs/general-usage...
Link to Channel's site:
https://hnawaz007.github.io/
--------------------------------------------------------------
💥Subscribe to our channel:
/ haqnawaz
📌 Links
-----------------------------------------
Follow me on social media!
🔗 GitHub: https://github.com/hnawaz007
📸 Instagram: / bi_insights_inc
📝 LinkedIn: / haq-nawaz
🔗 / hnawaz100
🚀 https://hnawaz007.github.io/
-----------------------------------------
#ETL #incremental #dlt
Topics in this video (click to jump around):
==================================
0:00 - Introduction to data load tool (dlt) incremental load
0:29 - Source Change Detection: Merge Write Disposition
1:27 - Source SQL Server Table Creation
2:02 - DLT Incremental Load Pipeline
2:48 - Test DLT Pipeline
3:26 - Add Trigger on Source Table
3:50 - Update/Insert records in Source SQL DB
4:11 - Run the dlt pipeline
4:20 - Review pipelines results
Watch video ETL Incremental Data Load Approach Using DLT | Source Change Detection | Load New & Change Data online without registration, duration hours minute second in high quality. This video was added by user BI Insights Inc 26 April 2024, don't forget to share it with your friends and acquaintances, it has been viewed on our site 1,11 once and liked it 3 people.