Monitor Apache Airflow with StatsD | Prometheus | Grafan | part1
#apacheairflow #airflowforbeginners #airflowmetrics #statsd #prometheus #grafana
In this video, you'll learn about how you can monitor airflow cluster by taking advantages of useful airflow statsd metrics. As part of single docker-compsoe file, we will spin up following services:
1. airflow-init: initializes airflow database schema and creates admin user
2. airflow-scheduler: runs airflow scheduler in LocalExecutor
3. airflow-webserver: runs airflow webserver exposing on port 8081
4. mysql: runs mysql database for airflow metadata
5. statsd-exporter: service that converts airflow statsd metrics to Prometheus metrics
6. Prometheus: runs Prometheus service that scraps airflow metrics from statsd-exporter
7. Grafana: runs Grafana instance where our final dashboards will be available.
---------------------------------------------------
👍 If this video was helpful to you, please don't forget to thumbs-up,
⌨️ leave a comment and
🔗 share with your friends.
⭐ Support our work by hitting that subscribe button.
🔔 hit bell icon to stay alert on more upcoming interesting stuff.
---------------------------------------------------
Timeline:
00:00 Intro
00:56 Monitoring data flow
01:41 Airflow metrics configs
02:01 Write docker-compose file
03:15 writing statsd-exporter service
04:02 statsd config mapping
06:01 test metrics
08:42 outro
-----------------------------------------------
Get complete code from here;
https://github.com/maxcotec/Apache-Ai...
Grafana and StatsD configs
https://github.com/databand-ai/airflo...
Statsd python client
https://github.com/jsocol/pystatsd
Watch the previous video here:
Run Airflow locally via Docker
• Build your first pipeline DAG | Apach...
Learn Something New Below:
https://maxcotec.com/blog/pv-center-f...
https://maxcotec.com/blog/5-cool-tech...
Watch video Monitor Apache Airflow with StatsD | Prometheus | Grafan | part1 online without registration, duration hours minute second in high quality. This video was added by user MaxcoTec Learning 23 September 2021, don't forget to share it with your friends and acquaintances, it has been viewed on our site 17,595 once and liked it 189 people.