How to Create a Dynamic Pipeline in Azure Synapse Analytics or Azure Data Factory!!
@SQLBalls walks us through how to create a dynamic pipeline in @AzureSynapse . This concept could also be used for @MSDataFactory , because they are so similar. He does this to move data from a database into @AzureDataLake in parquet files so he can use an @AzureSynapse Serverless SQL Pool to query his data.
Creating a Synapse workspace
https://learn.microsoft.com/en-us/azu...
Expressions and functions in Azure Data Factory and Azure Synapse Analytics
https://learn.microsoft.com/en-us/azu...
Pipeline execution and triggers in Azure Data Factory or Azure Synapse Analytics
https://learn.microsoft.com/en-us/azu...
Chapters
00:00 Introduction
00:42 Demo - Create Linked service
01:45 Demo - Add Azure Synapse System Assigned Managed Identity to Azure SQL DB
02:43 Demo - Build Pipeline
02:50 Demo - Dynamic Lookup Task
04:03 Demo - Dynamic ForEach
04:35 Demo - Dynamic Copy Data Task
08:43 Demo - Commit & Debug the Pipeline
09:12 Demo - Validate parquet output & Query with Serverless SQL Pool
09:40 DON'T FORGET TO UNCHECK THIS BOX!
/*******************
/CONNECT WITH US!!
/*******************
Tales from the Field
Twitter: / talesftfield
Linkedin: / 100091910
Blusky: https://bsky.app/profile/talesfromthe...
Bradley Ball
Twitter: / sqlballs
Linkedin: / sqlballs
Blusky : https://bsky.app/profile/sqlballs.bsk...
Daniel Taylor
Twitter: / dbabulldog
Linkedin: / danieltaylor3
Blusky: https://bsky.app/profile/dbabulldog.b...
Neeraj Jhaveri
Twitter: / neeraj_jhaveri
Linkedin: / neerajjhaveri
Blusky: https://bsky.app/profile/neerajny.bsk...
Josh Luedeman
Twitter: / joshluedeman
Linkedin: / joshluedeman
Blusky: https://bsky.app/profile/joshluedeman...
Andrés Padilla
Twitter: / nodestreamio
Linkedin: / andrespadillaandrade
Watch video How to Create a Dynamic Pipeline in Azure Synapse Analytics or Azure Data Factory!! online without registration, duration hours minute second in high quality. This video was added by user Tales From The Field 03 May 2023, don't forget to share it with your friends and acquaintances, it has been viewed on our site 2,005 once and liked it 47 people.