Dynamic Databricks Workflows - Advancing Spark
Last week saw the General Availability of dynamic functionality for Databricks Workflows, in the form of the parameterized ForEach activity, but what does that mean? And why should we care?
For a long time we've been using external orchestration tools whenever things had to be flexible, metadata driven or simply at a large scale - but with these changes we have a cheap, flexible way of achieving some pretty complex orchestration inside of Databricks itself!
In this video Simon looks at a quick pattern for running a medallion-style lake processing routine, all driven by some JSON metadata!
For the release announcements, see: https://www.databricks.com/blog/strea...
Watch video Dynamic Databricks Workflows - Advancing Spark online without registration, duration hours minute second in high quality. This video was added by user Advancing Analytics 03 September 2024, don't forget to share it with your friends and acquaintances, it has been viewed on our site 4,53 once and liked it 9 people.