Realtime Streaming with Kafka and Telegram | End to End Data Engineering Project
In this comprehensive tutorial, you will build an end-to-end data engineering pipeline for real-time YouTube Analytics. Each time there's an activity on any video you or playlist of your choice, you get instant notification on Telegram.
📋 What You Will Learn:
✅ How to fetch data from YouTube API using Python
✅ Setting up a Kafka ecosystem using Docker and Confluent containers
✅ Processing and streaming data using ksqlDB
✅ Sending data to external systems with connectors
✅ Real-time notifications on Telegram
✅ Python Advanced Concepts
✅ Google cloud configuration for Youtube
🛠 Technologies Used:
✅ Python
✅ Google Cloud
✅ Docker
✅ Telegram
🔗 Useful Links:
✅ GitHub Repo for this Project: https://github.com/airscholar/YoutubeAnalytics.git
✅ Confluent Official Documentation: https://docs.confluent.io/home/overview.html
✅ YouTube API Documentation: https://developers.google.com/youtube/v3/docs
✅ Kafka-Python Documentation: https://kafka-python.readthedocs.io/en/master/
🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
🗨 Comments & Questions
Have questions or ran into issues? Drop a comment below and I'll do my best to help you out!
TIMESTAMPS:
0:00 Introduction
2:21 Setting up the system architecture on Docker
17:46 Control Center Demo
23:20 Getting Youtube API Key from Google Cloud
27:21 Fetching Data From Youtube with Python
37:53 Streaming Data to Kafka
45:13 Advanced Python Concept
59:31 Stream Processing with KSQLDB
1:09:50 Setting up Telegram Bot
1:13:08 Connecting to external systems from Kafka (Telegram)
1:27:18 Outro
✨ Tags ✨
Data Engineering, Kafka, Zookeeper, Docker, Docker Compose, ETL Pipeline, Data Pipeline, Big Data, Streaming Data, Real-time Analytics, Kafka Connect, Schema Registry, Control Center, Data Streaming, Google Cloud, Youtube API, Telegram, KSqlDb, Confluent Connect
✨ Hashtags ✨
#confluent #DataEngineering #ApacheAirflow #Kafka #telegram #PostgreSQL #Docker #ETLPipeline #DataPipeline #StreamingData #RealTimeAnalytics #docker #python #java
-
20:37
ROSE UNPLUGGED
9 hours ago4th of July Message and Kelsey Grammer Interview
5.61K2 -
50:06
The Charlie Kirk Show
2 hours agoThe Secret History of Communist Revolutions ft. Jack Posobiec and Joshua Lisec
27.3K10 -
1:49:48
Glenn Greenwald
5 hours agoPresidential Immunity's Long History; Hysteria Radically Distorts SCOTUS Ruling; Dem Oligarchs Forcing Biden Out of 2024 Race | SYSTEM UPDATE #292
60K113 -
57:12
Havoc
5 hours agoCelebrating Independent Artists | Stuck Off the Realness Ep. 3
27.2K1 -
1:14:02
Donald Trump Jr.
11 hours agoMedia Plays Dumb on Biden Decline, Interview with Alex Marlow! | TRIGGERED Ep.151
90.1K96 -
1:47:01
2 MIKES LIVE
5 hours ago2 MIKES LIVE #86 NEWS BREAKDOWN WEDNESDAY, WITH SPECIAL GUEST JOY PULLMANN!
29.3K1 -
2:28:04
WeAreChange
5 hours agoBiden BANKROLLERS Want Return On THEIR INVESTMENT, Order JOE TO GO!
48.3K17 -
16:24
China Uncensored
4 hours agoChina's Economy is in Ruins—Local Debt Out of Control
24.2K59 -
13:40
Brad Polumbo
12 hours agoWoke TikTokers LOSE THEIR MIND Over Conservative Supreme Court Win!
37.7K23 -
1:14:46
ReasonTV
11 hours agoAndy Mills: Quitting The New York Times and making 'The Witch Trials of J.K. Rowling'
26.8K2