Realtime Streaming with Kafka and Telegram | End to End Data Engineering Project

Save
1 year ago
25
Data Engineering Kafka Zookeeper Docker Docker Compose ETL Pipeline Data Pipeline Big Data Streaming Data Real-time Analytics Kafka Connect

In this comprehensive tutorial, you will build an end-to-end data engineering pipeline for real-time YouTube Analytics. Each time there's an activity on any video you or playlist of your choice, you get instant notification on Telegram.

📋 What You Will Learn:
✅ How to fetch data from YouTube API using Python
✅ Setting up a Kafka ecosystem using Docker and Confluent containers
✅ Processing and streaming data using ksqlDB
✅ Sending data to external systems with connectors
✅ Real-time notifications on Telegram
✅ Python Advanced Concepts
✅ Google cloud configuration for Youtube

🛠 Technologies Used:
✅ Python
✅ Google Cloud
✅ Docker
✅ Telegram

🔗 Useful Links:
✅ GitHub Repo for this Project: https://github.com/airscholar/YoutubeAnalytics.git
✅ Confluent Official Documentation: https://docs.confluent.io/home/overview.html
✅ YouTube API Documentation: https://developers.google.com/youtube/v3/docs
✅ Kafka-Python Documentation: https://kafka-python.readthedocs.io/en/master/

🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟

🗨 Comments & Questions
Have questions or ran into issues? Drop a comment below and I'll do my best to help you out!

TIMESTAMPS:
0:00 Introduction
2:21 Setting up the system architecture on Docker
17:46 Control Center Demo
23:20 Getting Youtube API Key from Google Cloud
27:21 Fetching Data From Youtube with Python
37:53 Streaming Data to Kafka
45:13 Advanced Python Concept
59:31 Stream Processing with KSQLDB
1:09:50 Setting up Telegram Bot
1:13:08 Connecting to external systems from Kafka (Telegram)
1:27:18 Outro

✨ Tags ✨
Data Engineering, Kafka, Zookeeper, Docker, Docker Compose, ETL Pipeline, Data Pipeline, Big Data, Streaming Data, Real-time Analytics, Kafka Connect, Schema Registry, Control Center, Data Streaming, Google Cloud, Youtube API, Telegram, KSqlDb, Confluent Connect

✨ Hashtags ✨
#confluent #DataEngineering #ApacheAirflow #Kafka #telegram #PostgreSQL #Docker #ETLPipeline #DataPipeline #StreamingData #RealTimeAnalytics #docker #python #java

Loading comments...