Dynamic Databricks Workflows - Advancing Spark
Last week saw the General Availability of dynamic functionality for Databricks Workflows, in the form of the parameterized ForEach activity, but what does that mean? And why should we care?
For a long time we've been using external orchestration tools whenever things had to be flexible, metadata driven or simply at a large scale - but with these changes we have a cheap, flexible way of achieving some pretty complex orchestration inside of Databricks itself!
In this video Simon looks at a quick pattern for running a medallion-style lake processing routine, all driven by some JSON metadata!
For the release announcements, see: https://www.databricks.com/blog/strea...
Смотрите видео Dynamic Databricks Workflows - Advancing Spark онлайн без регистрации, длительностью часов минут секунд в хорошем качестве. Это видео добавил пользователь Advancing Analytics 03 Сентябрь 2024, не забудьте поделиться им ссылкой с друзьями и знакомыми, на нашем сайте его посмотрели 4,53 раз и оно понравилось 9 людям.