Realistic ML Example With Dagster
Here's a quick video of a much more realistic #ml pipeline using #dagster written by Jordan Volz
If you're new to dagster, it's worth checking out as a way to orchestrate ML. Hopefully this quick harbor tour provides a basic intro.
And while there are many good reasons to use an orchestration tool, here's a simple one from my experience running this specific project:
I made a mistake when configuring this project on an EC2 instance.
Thanks to dagster, I could see exactly which step failed
One can always test the code directly, but for me, it sure was easier to click around in the UI and see the exact error message. This made debugging faster, e.g., slack jordan with my error message, make the change he suggested, click a button in the UI to test :)
Eventually, sharing the dagit UI with him so he could poke around and re-run stuff
Ultimately we got everything working - go figure the computer wasn't wrong, it was a configuration mistake I made, but dagster made it easier to work cough ask for help
As an aside, if you're using dagster to orchestrate your ML pipelines, we'd love to chat with you. Feel free to connect with me on LI: / gustafrcavanaugh
Watch video Realistic ML Example With Dagster online without registration, duration hours minute second in high quality. This video was added by user Gus Cavanaugh 08 December 2022, don't forget to share it with your friends and acquaintances, it has been viewed on our site 563 once and liked it 0 people.