Premium Only Content

Realtime Streaming with Apache Flink | End to End Data Engineering Project
In this video, you will be building an end-to-end data engineering project using some of the most powerful technologies in the industry: Apache Flink, Kafka, Elasticsearch, and Docker. In this video, we dive deep into the world of real-time data processing and analytics, guiding you through every step of creating a robust, scalable data pipeline.
Timestamp
0:00 Introduction
0:55 The system architecture
08:00 Sales Analytics Data Generation
19:10 Producing Data into Kafka Broker
25:00 Setting up Apache Flink project
32:28 Consuming data from Kafka with Apache Flink
43:30 Starting Apache Flink on Mac
54:25 Writing Kafka Streams to Postgres Database
1:20:00 Aggregating Transactions per Category into Postgres
1:36:00 Aggregating Transactions Per Day into Postgres
1:39:46 Aggregating Transactions Per Month into Postgres
1:51:52 Writing Kafka Streams Data into Elasticsearch
2:05:00 Reindexing Data on Elasticsearch with Timestamp
2:10:52 Creating Streaming Dashboard on Elasticsearch
2:22:46 Realtime Dashboard Results
2:24:14 Recap
2:25:34 Outro
👦🏻 My Linkedin: https://www.linkedin.com/in/yusuf-ganiyu-b90140107/
🚀 Twitter: https://twitter.com/YusufOGaniyu
📝 Medium: https://medium.com/@yusuf.ganiyu
🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
🔗 Useful Links and Resources:
✅ Code: https://github.com/airscholar/FlinkCommerce.git
✅ Medium Article: https://medium.com/@yusuf.ganiyu/realtime-data-engineering-project-with-airflow-kafka-spark-cassandra-and-postgres-804bcd963974
✅ Docker Compose Documentation: https://docs.docker.com/compose/
✅ Apache Kafka Official Site: https://kafka.apache.org/
✅ Apache Flink Official Documentation: https://nightlies.apache.org/flink/flink-docs-stable/
✅ Confluent Docs: https://docs.confluent.io/home/overview.html
✅ Maven Repository: https://mvnrepository.com/
✨ Tags ✨
Big Data Engineering, Apache Flink, Kafka, Elasticsearch, Docker, Data Engineering, Realtime Data Processing, Big Data, Data Pipeline, Streaming Data, Data Analytics, Tech Tutorial, Data Science, Flink Streaming, Kafka Streaming, Elasticsearch Tutorial, Docker Containers, Data Engineering Project, Realtime Analytics, Big Data Technologies, Data Engineering Tutorial, Data Engineering Projects, Data Engineer
✨Hashtags✨
#ApacheFlink, #Kafka, #Elasticsearch, #Docker, #DataEngineering, #RealtimeData, #BigData, #DataPipeline, #TechTutorial, #DataScience, #StreamingData, #Flink, #KafkaStreams, #ElasticsearchTips, #DockerContainers, #DataEngineeringProjects, #RealtimeAnalytics, #BigDataTech, #LearnDataEngineering, #dataengineers
-
2:04:11
MattMorseTV
8 hours ago $15.34 earned🔴The UK just hit ROCK BOTTOM.🔴
70.6K158 -
iCheapshot
5 hours ago $2.02 earnedBack From Vacation! | Checking Out Skate Maybe Some Boarderlands 4
58.8K1 -
3:08:37
Jorba4
5 hours ago🔴Live-Jorba4- Borderlands w/ The TRIBE
48.6K -
54:43
Glenn Greenwald
8 hours agoGlenn Reacts to Netanyahu's UN Speech; PLUS: Q&A on Trump's Russia/Ukraine Policy, the Tom Homan Investigation, and More | SYSTEM UPDATE #522
87.5K48 -
43:54
Donald Trump Jr.
7 hours agoJames and the Giant Breach, Plus Explosive J6 Revelations | TRIGGERED Ep.277
171K186 -
4:45:01
MadHouse_
7 hours agoFRIEND FRIDAY! - Garys Mod ? im pretty sure im gonna lose at this game
31.9K5 -
2:26:51
blackfox87
6 hours agoBorderlands Grind Continues | PREMIUM CREATOR | #DisabledVeteran
29.6K1 -
4:30:11
Mally_Mouse
5 days agoFriend Friday!! 🎉 - Let's Play! - Gary's Mod!
19.4K -
10:17:07
ZWOGs
13 hours ago🔴LIVE IN 1440p! - Resident Evil 3 First Playthrough, GMod w/ Mally_Mouse & Friends - Come Hang Out!
13.1K3 -
1:00:55
BonginoReport
11 hours agoJames Comey Indicted! - Nightly Scroll w/ Hayley Caronia (Ep.143)
150K124