Premium Only Content

Smart City End to End Realtime Data Engineering Project | Get Hired as an AWS Data Engineer
In this video, you will be building a Smart City End to End Realtime data streaming pipeline covering each phase from data ingestion to processing and finally storage. We'll utilize tools like IOT devices, Apache Zookeeper, Apache Kafka, Apache Spark, Docker, Python, AWS Cloud, AWS Glue, AWS Athena, AWS IAM, AWS Redshift and finally PowerBI to visualize data on Reshift.
Like this video?
- Buy me a coffee: https://www.buymeacoffee.com/yusuf.ganiyu
- Become a member: https://www.youtube.com/@codewithyu/join
Timestamps:
0:00 Introduction
1:29 System Architecture
7:22 Project Setup
9:00 Docker containers setup and coding
26:17 IOT services producer
38:19 Vehicle information Generator
48:10 GPS Information Generator
50:13 Traffic information Generator
53:13 Weather information Generator
58:35 Emergency Incident Generator
1:03:39 Producing IOT Data to Kafka
1:14:43 AWS S3 setup with policies
1:16:38 AWS IAM Roles and Credentials Management
1:19:14 Apache Spark Realtime Streaming from Kafka
2:01:14 Fixing Schema Issues in Apache Spark Structured Streaming
2:07:31 AWS Glue Crawlers
2:10:23 Working with AWS Athena
2:13:22 Loading Data into Redshift from AWS Glue Data Catalog
2:17:58 Connecting and Querying Redshift DW with DBeaver
2:20:51 Connecting Redshift to AWS Glue Catalog
2:23:34 Fixing IAM Permission issues with Redshift
2:26:05 Outro
👦🏻 My Linkedin: https://www.linkedin.com/in/yusuf-ganiyu-b90140107/
🚀 X(Twitter): https://x.com/YusufOGaniyu
📝 Medium: https://medium.com/@yusuf.ganiyu
🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
🔗 Useful Links and Resources:
✅ Docker Compose Documentation: https://docs.docker.com/compose/
✅ Apache Kafka Official Site: https://kafka.apache.org/
✅ Apache Spark Official Site: https://spark.apache.org/
✅ Confluent Docs: https://docs.confluent.io/home/overview.html
✅ S3 Documentation: https://docs.aws.amazon.com/s3/
✅ AWS IAM Documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html
✨ Tags ✨
Data Engineering, Apache Airflow, Kafka, Apache Spark, Cassandra, PostgreSQL, Zookeeper, Docker, Docker Compose, ETL Pipeline, Data Pipeline, Big Data, Streaming Data, Real-time Analytics, Kafka Connect, Spark Master, Spark Worker, Schema Registry, Control Center, Data Streaming
✨ Hashtags ✨
#confluent #DataEngineering #ApacheAirflow #Kafka #ApacheSpark #Cassandra #PostgreSQL #Docker #ETLPipeline #DataPipeline #StreamingData #RealTimeAnalytics
-
LIVE
blackfox87
47 minutes agoFoxyFam takes on Warzone! | PREMIUM CREATOR | #DisabledVeteran
54 watching -
LIVE
Spartan
4 hours agoScrims then Ranked / Octopath Traveler 2
30 watching -
LIVE
The Jimmy Dore Show
2 hours agoTrump Administration Sends Accused Pedo BACK TO ISRAEL! Ukrainians Now OVERWHELMINGLY Oppose War!
9,110 watching -
6:44:51
Dr Disrespect
9 hours ago🔴LIVE - DR DISRESPECT - IMPOSSIBLE 5 CHICKEN DINNER CHALLENGE - FEAT. VISS
91.2K15 -
LIVE
GloryJean
1 hour agoDominating The Sniper Role 🖱️ 6.7 K/D | Duos w/ Spartakus
19 watching -
LIVE
BigTallRedneck
1 hour agoBRRRAP PACK VS ANYBODY!!
42 watching -
1:09:21
TheCrucible
4 hours agoThe Extravaganza! Ep. 24 (8/20/25)
56K9 -
1:18:42
Kim Iversen
4 hours agoUFO Base Area 51 Catches Fire… Is It a Massive Cover-Up?!
29.5K57 -
1:51:18
Redacted News
4 hours ago"There will be consequences!!!" Trump issues big threat to Putin ahead of peace summit | Redacted
97.8K90 -
53:14
Candace Show Podcast
4 hours agoThe MOST MORAL Blackmail In The World | Candace EP 231
51.4K143