Premium Only Content

Smart City End to End Realtime Data Engineering Project | Get Hired as an AWS Data Engineer
In this video, you will be building a Smart City End to End Realtime data streaming pipeline covering each phase from data ingestion to processing and finally storage. We'll utilize tools like IOT devices, Apache Zookeeper, Apache Kafka, Apache Spark, Docker, Python, AWS Cloud, AWS Glue, AWS Athena, AWS IAM, AWS Redshift and finally PowerBI to visualize data on Reshift.
Like this video?
- Buy me a coffee: https://www.buymeacoffee.com/yusuf.ganiyu
- Become a member: https://www.youtube.com/@codewithyu/join
Timestamps:
0:00 Introduction
1:29 System Architecture
7:22 Project Setup
9:00 Docker containers setup and coding
26:17 IOT services producer
38:19 Vehicle information Generator
48:10 GPS Information Generator
50:13 Traffic information Generator
53:13 Weather information Generator
58:35 Emergency Incident Generator
1:03:39 Producing IOT Data to Kafka
1:14:43 AWS S3 setup with policies
1:16:38 AWS IAM Roles and Credentials Management
1:19:14 Apache Spark Realtime Streaming from Kafka
2:01:14 Fixing Schema Issues in Apache Spark Structured Streaming
2:07:31 AWS Glue Crawlers
2:10:23 Working with AWS Athena
2:13:22 Loading Data into Redshift from AWS Glue Data Catalog
2:17:58 Connecting and Querying Redshift DW with DBeaver
2:20:51 Connecting Redshift to AWS Glue Catalog
2:23:34 Fixing IAM Permission issues with Redshift
2:26:05 Outro
👦🏻 My Linkedin: https://www.linkedin.com/in/yusuf-ganiyu-b90140107/
🚀 X(Twitter): https://x.com/YusufOGaniyu
📝 Medium: https://medium.com/@yusuf.ganiyu
🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
🔗 Useful Links and Resources:
✅ Docker Compose Documentation: https://docs.docker.com/compose/
✅ Apache Kafka Official Site: https://kafka.apache.org/
✅ Apache Spark Official Site: https://spark.apache.org/
✅ Confluent Docs: https://docs.confluent.io/home/overview.html
✅ S3 Documentation: https://docs.aws.amazon.com/s3/
✅ AWS IAM Documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html
✨ Tags ✨
Data Engineering, Apache Airflow, Kafka, Apache Spark, Cassandra, PostgreSQL, Zookeeper, Docker, Docker Compose, ETL Pipeline, Data Pipeline, Big Data, Streaming Data, Real-time Analytics, Kafka Connect, Spark Master, Spark Worker, Schema Registry, Control Center, Data Streaming
✨ Hashtags ✨
#confluent #DataEngineering #ApacheAirflow #Kafka #ApacheSpark #Cassandra #PostgreSQL #Docker #ETLPipeline #DataPipeline #StreamingData #RealTimeAnalytics
-
LIVE
DoldrumDan
2 hours agoCHALLENGE RUNNER BOUT DONE WITH ELDEN RING NIGHTREIGN STORY MODE HUGE GAMING
97 watching -
10:59
itsSeanDaniel
1 day agoEuropean Leaders INSTANTLY REGRET Disrespecting Trump
19.6K16 -
16:43
GritsGG
17 hours agoThey Buffed This AR & It Slaps! Warzone Loadout!
18.1K1 -
2:05:30
Side Scrollers Podcast
21 hours agoEveryone Hates MrBeast + FBI Spends $140k on Pokemon + All Todays News | Side Scrollers Live
114K12 -
11:06
The Pascal Show
15 hours ago $1.47 earned'THEY'RE GETTING DEATH THREATS!' Jake Haro's Lawyer Breaks Silence On Emmanuel Haro's Disappearance!
18.6K2 -
LIVE
Lofi Girl
2 years agoSynthwave Radio 🌌 - beats to chill/game to
295 watching -
2:19:32
Badlands Media
1 day agoDEFCON ZERO Ep. 005: False Flags, Cyber Fronts & Global Power Plays
159K74 -
2:35:23
FreshandFit
10 hours agoWhy Black Men Don't Date Black Women Debate
46.5K49 -
2:03:42
Inverted World Live
14 hours agoBigfoot Corpse Coming to the NY State Fair | Ep. 94
113K27 -
6:16:23
SpartakusLIVE
15 hours ago$1,000 Pistol Challenge || #1 ENTERTAINER of The EONS Eradicates BOREDOM
88.5K2