Cricket Statistics Data Pipeline in Google Cloud using Airflow | Data Engineering Project
Looking to get in touch?
Drop me a line at [email protected], or schedule a meeting using the provided link https://topmate.io/vishal_bulbule
Cricket Statistics Data Pipeline in Google Cloud using Airflow,Dataflow,Cloud Function and Looker Studio
Data Retrieval: We fetch data from the Cricbuzz API using Python.
Storing Data in GCS: After fetching the data, we store it in a CSV file in Google Cloud Storage (GCS).
Cloud Function Trigger: Create a Cloud Function that triggers upon file upload to the GCS bucket. The function will execute when a new CSV file is detected and trigger dataflow job.
Cloud Function Execution: Inside the Cloud Function, we will have code that triggers a Dataflow job. Ensure you handle the trigger correctly and pass the required parameters to initiate the Dataflow job.
Dataflow Job: The Dataflow job is triggered by the Cloud Function and loads the data from the CSV file in the GCS bucket into BigQuery. Ensure you have set up the necessary configurations.
Looker Dashboard: BigQuery serves as the data source for your Looker Studio dashboard. Configure Looker to connect to BigQuery and create the dashboard based on the data loaded.
Github Repo for all code used in this project
https://github.com/vishal-bulbule/cri...
============================================
Associate Cloud Engineer -Complete Free Course
• Associate Cloud Engineer -Complete Fr...
Google Cloud Data Engineer Certification Course
• Google Cloud Data Engineer Certificat...
Google Cloud Platform(GCP) Tutorials
• Google Cloud Platform(GCP) Tutorials
Generative AI
• Generative AI
Getting Started with Duet AI
• Getting started with Duet AI | Google...
Google Cloud Projects
• Google Cloud Projects
Python For GCP
• Python for GCP
Terraform Tutorials
• Terraform Associate Certification(00...
Linkedin
/ vishal-bulbule
Medium Blog
/ vishalbulbule
Github Repository for Source Code
https://github.com/vishal-bulbule
Email - [email protected]
#dataengineeringessentials #dataengineers #dataengineeringproject #airflow #dataflow #cloudcomposer #bigquery #looker #googlecloud #datapipeline
Смотрите видео Cricket Statistics Data Pipeline in Google Cloud using Airflow | Data Engineering Project онлайн без регистрации, длительностью часов минут секунд в хорошем качестве. Это видео добавил пользователь TechTrapture 18 Декабрь 2023, не забудьте поделиться им ссылкой с друзьями и знакомыми, на нашем сайте его посмотрели 20,376 раз и оно понравилось 448 людям.