Premium Only Content

Realtime Data Streaming | End To End Data Engineering Project
In this video, you will be building a real-time data streaming pipeline, covering each phase from data ingestion to processing and finally storage. We'll utilize a powerful stack of tools and technologies, including Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra—all neatly containerized using Docker.
📚 What You'll Learn:
👉 Setting up a data pipeline with Apache Airflow
👉 Streaming data with Kafka and Kafka Connect
👉 Using Zookeeper for distributed synchronization
👉 Data processing with Apache Spark
👉 Data storage solutions with Cassandra and PostgreSQL
👉 Containerizing your data engineering environment with Docker
✨ Timestamps: ✨
0:00 Introduction
0:53 System architecture
3:47 Getting data from API with Airflow
17:10 Docker Compose for the architecture
26:09 Streaming data into Kafka
44:29 Apache Spark and Cassandra setup
49:33 Streaming data into cassandra
1:27:05 Outro
👦🏻 My Linkedin: https://www.linkedin.com/in/yusuf-ganiyu-b90140107/
🚀 Twitter: https://twitter.com/YusufOGaniyu
📝 Medium: https://medium.com/@yusuf.ganiyu
🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
🔗 Useful Links and Resources:
✅ Code: https://github.com/airscholar/e2e-data-engineering.git
✅ Medium Article: https://medium.com/@yusuf.ganiyu/realtime-data-engineering-project-with-airflow-kafka-spark-cassandra-and-postgres-804bcd963974
✅ Docker Compose Documentation: https://docs.docker.com/compose/
✅ Apache Kafka Official Site: https://kafka.apache.org/
✅ Apache Spark Official Site: https://spark.apache.org/
✅ Apache Airflow Official Site: https://airflow.apache.org/
✅ Cassandra: https://cassandra.apache.org/
✅ Confluent Docs: https://docs.confluent.io/home/overview.html
✨ Tags ✨
Data Engineering, Apache Airflow, Kafka, Apache Spark, Cassandra, PostgreSQL, Zookeeper, Docker, Docker Compose, ETL Pipeline, Data Pipeline, Big Data, Streaming Data, Real-time Analytics, Kafka Connect, Spark Master, Spark Worker, Schema Registry, Control Center, Data Streaming
✨ Hashtags ✨
#confluent #DataEngineering #ApacheAirflow #Kafka #ApacheSpark #Cassandra #PostgreSQL #Docker #ETLPipeline #DataPipeline #StreamingData #RealTimeAnalytics
-
2:04:11
MattMorseTV
8 hours ago $15.34 earned🔴The UK just hit ROCK BOTTOM.🔴
70.6K158 -
iCheapshot
5 hours ago $2.02 earnedBack From Vacation! | Checking Out Skate Maybe Some Boarderlands 4
58.8K1 -
3:08:37
Jorba4
5 hours ago🔴Live-Jorba4- Borderlands w/ The TRIBE
48.6K -
54:43
Glenn Greenwald
8 hours agoGlenn Reacts to Netanyahu's UN Speech; PLUS: Q&A on Trump's Russia/Ukraine Policy, the Tom Homan Investigation, and More | SYSTEM UPDATE #522
87.5K48 -
43:54
Donald Trump Jr.
7 hours agoJames and the Giant Breach, Plus Explosive J6 Revelations | TRIGGERED Ep.277
171K186 -
4:45:01
MadHouse_
7 hours agoFRIEND FRIDAY! - Garys Mod ? im pretty sure im gonna lose at this game
31.9K5 -
2:26:51
blackfox87
6 hours agoBorderlands Grind Continues | PREMIUM CREATOR | #DisabledVeteran
27.9K1 -
4:30:11
Mally_Mouse
5 days agoFriend Friday!! 🎉 - Let's Play! - Gary's Mod!
19.4K -
10:17:07
ZWOGs
13 hours ago🔴LIVE IN 1440p! - Resident Evil 3 First Playthrough, GMod w/ Mally_Mouse & Friends - Come Hang Out!
13.1K3 -
1:00:55
BonginoReport
11 hours agoJames Comey Indicted! - Nightly Scroll w/ Hayley Caronia (Ep.143)
150K124