Airflow sensors | Create custom sensor to sense files
See how airflow sensors can pitch in your ETL pipelines to sense something before proceeding with downstream dependencies.
Get code here:
https://github.com/maxcotec/Apache-Ai...
How to deploy airflow locally using Docker
• Run Airflow 2.0 via Docker | Minimal ...
How to deploy airflow locally on a multinode kubernets cluster
• Airflow with Kubernetes Executors | R...
build your first data pipeline in airflow
• Build your first pipeline DAG | Apach...
#apacheairflow #kubernetes #airflowsensors #dataengineering #etl #maxcoteclearning
Video outline:
00:00 intro and outline
00:38 definition
01:27 operation modes
02:23 sensor types
02:57 github repo & reference videos
03:47 spin-up airflow locally
05:15 write dag with BashSensor
06:42 test BashSensor
07:18 write custom sensor - check file exists
09:19 test custom file-check sensor
10:30 outro
Watch video Airflow sensors | Create custom sensor to sense files online without registration, duration hours minute second in high quality. This video was added by user MaxcoTec Learning 22 July 2023, don't forget to share it with your friends and acquaintances, it has been viewed on our site 1,723 once and liked it 21 people.