Apache Airflow | Build Custom Sensor | KubernetesPodSensor
In this video we will build a custom sensor for Apache Airflow known as KubernetesPodSensor. This Sensor is inspired from KubernetesPodOperator, which lets you run a docker image in a dedicated pod on each poke interval.
Prerequsites:
1. Airflow Sensors: • Airflow sensors | Create custom senso...
2. Airflow on Kubernetes: • Airflow with Kubernetes Executors | R...
2.1 Airflow on Kubernetes using rancher: • Run Airflow on kubernetes using Ranch...
3. Airflow KubernetesPodOperators: • Deep dive into Airflow Kubernetes Pod...
Get the code:
dynamoDB-readiness-check: https://github.com/maxcotec/dynamodb-...
dag and sensor code: https://github.com/maxcotec/airflow-k...
Timeline:
00:00 intro & outline
00:38 prerequisites
01:31 Airflow Sensor recap
02:08 why KubernetesPodSensor
02:43 How KubernetesPodSensor will work
04:13 write code for sensor
06:28 write dag code
09:23 build image
11:06 deploy airflow
12:07 test the dag
15:02 conclusion
15:53 outro
#apacheairflow #maxcoteclearning #apacheairflowsensor
Watch video Apache Airflow | Build Custom Sensor | KubernetesPodSensor online without registration, duration hours minute second in high quality. This video was added by user MaxcoTec Learning 01 January 1970, don't forget to share it with your friends and acquaintances, it has been viewed on our site 717 once and liked it 8 people.