Premium Only Content

Realtime Streaming with Apache Flink | End to End Data Engineering Project
In this video, you will be building an end-to-end data engineering project using some of the most powerful technologies in the industry: Apache Flink, Kafka, Elasticsearch, and Docker. In this video, we dive deep into the world of real-time data processing and analytics, guiding you through every step of creating a robust, scalable data pipeline.
Timestamp
0:00 Introduction
0:55 The system architecture
08:00 Sales Analytics Data Generation
19:10 Producing Data into Kafka Broker
25:00 Setting up Apache Flink project
32:28 Consuming data from Kafka with Apache Flink
43:30 Starting Apache Flink on Mac
54:25 Writing Kafka Streams to Postgres Database
1:20:00 Aggregating Transactions per Category into Postgres
1:36:00 Aggregating Transactions Per Day into Postgres
1:39:46 Aggregating Transactions Per Month into Postgres
1:51:52 Writing Kafka Streams Data into Elasticsearch
2:05:00 Reindexing Data on Elasticsearch with Timestamp
2:10:52 Creating Streaming Dashboard on Elasticsearch
2:22:46 Realtime Dashboard Results
2:24:14 Recap
2:25:34 Outro
👦🏻 My Linkedin: https://www.linkedin.com/in/yusuf-ganiyu-b90140107/
🚀 Twitter: https://twitter.com/YusufOGaniyu
📝 Medium: https://medium.com/@yusuf.ganiyu
🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
🔗 Useful Links and Resources:
✅ Code: https://github.com/airscholar/FlinkCommerce.git
✅ Medium Article: https://medium.com/@yusuf.ganiyu/realtime-data-engineering-project-with-airflow-kafka-spark-cassandra-and-postgres-804bcd963974
✅ Docker Compose Documentation: https://docs.docker.com/compose/
✅ Apache Kafka Official Site: https://kafka.apache.org/
✅ Apache Flink Official Documentation: https://nightlies.apache.org/flink/flink-docs-stable/
✅ Confluent Docs: https://docs.confluent.io/home/overview.html
✅ Maven Repository: https://mvnrepository.com/
✨ Tags ✨
Big Data Engineering, Apache Flink, Kafka, Elasticsearch, Docker, Data Engineering, Realtime Data Processing, Big Data, Data Pipeline, Streaming Data, Data Analytics, Tech Tutorial, Data Science, Flink Streaming, Kafka Streaming, Elasticsearch Tutorial, Docker Containers, Data Engineering Project, Realtime Analytics, Big Data Technologies, Data Engineering Tutorial, Data Engineering Projects, Data Engineer
✨Hashtags✨
#ApacheFlink, #Kafka, #Elasticsearch, #Docker, #DataEngineering, #RealtimeData, #BigData, #DataPipeline, #TechTutorial, #DataScience, #StreamingData, #Flink, #KafkaStreams, #ElasticsearchTips, #DockerContainers, #DataEngineeringProjects, #RealtimeAnalytics, #BigDataTech, #LearnDataEngineering, #dataengineers
-
38:41
JohnXSantos
23 hours ago $0.56 earnedHow To Start A CLOTHING BRAND on a BUDGET! Step X Step (2025)
6.75K1 -
1:26:34
Man in America
19 hours agoExposing the Cover-Up That Could Collapse Big Medicine: Parasites
70.7K63 -
30:57
Her Patriot Voice
16 hours ago $15.61 earnedDemocrats More Unhinged Than EVER Before!
93.3K127 -
29:13
Clownfish TV
1 day agoGen Z are Becoming the Boomers?! | Clownfish TV
15.9K35 -
1:48:31
Squaring The Circle, A Randall Carlson Podcast
19 hours agoMEGA Tsunamis and the formation of our World ft. Dr. Dallas Abbot
31.2K7 -
29:26
Advanced Level Diagnostics
2 days ago $0.12 earned2019 Chevy Express - No Crank, Relay Clicking! Diag & Fix!
3.97K -
30:56
5AMPodcast
23 hours ago $0.20 earnedCitizen Journalism 🎙️Replacing Traditional Media | Sam Anthony on 5 AM Podcast
5.23K1 -
6:53
Rena Malik, M.D.
1 day ago $2.05 earnedWhy Antidepressants Wreak Havoc on Your Sex Life?! | Urologist Explains How to Boost your Libido
25.3K4 -
1:00:00
BEK TV
2 days agoMIKE MOTSCHENBACHER ON NORTH DAKOTA POLITICS, TEA PARTY ROOTS, AND THE 2026 ELECTION
18.9K -
15:31
Breaking Points
1 day agoIs Trump Planning VENEZUELA Regime Change?
43.2K30