Premium Only Content

Smart City End to End Realtime Data Engineering Project | Get Hired as an AWS Data Engineer
In this video, you will be building a Smart City End to End Realtime data streaming pipeline covering each phase from data ingestion to processing and finally storage. We'll utilize tools like IOT devices, Apache Zookeeper, Apache Kafka, Apache Spark, Docker, Python, AWS Cloud, AWS Glue, AWS Athena, AWS IAM, AWS Redshift and finally PowerBI to visualize data on Reshift.
Like this video?
- Buy me a coffee: https://www.buymeacoffee.com/yusuf.ganiyu
- Become a member: https://www.youtube.com/@codewithyu/join
Timestamps:
0:00 Introduction
1:29 System Architecture
7:22 Project Setup
9:00 Docker containers setup and coding
26:17 IOT services producer
38:19 Vehicle information Generator
48:10 GPS Information Generator
50:13 Traffic information Generator
53:13 Weather information Generator
58:35 Emergency Incident Generator
1:03:39 Producing IOT Data to Kafka
1:14:43 AWS S3 setup with policies
1:16:38 AWS IAM Roles and Credentials Management
1:19:14 Apache Spark Realtime Streaming from Kafka
2:01:14 Fixing Schema Issues in Apache Spark Structured Streaming
2:07:31 AWS Glue Crawlers
2:10:23 Working with AWS Athena
2:13:22 Loading Data into Redshift from AWS Glue Data Catalog
2:17:58 Connecting and Querying Redshift DW with DBeaver
2:20:51 Connecting Redshift to AWS Glue Catalog
2:23:34 Fixing IAM Permission issues with Redshift
2:26:05 Outro
👦🏻 My Linkedin: https://www.linkedin.com/in/yusuf-ganiyu-b90140107/
🚀 X(Twitter): https://x.com/YusufOGaniyu
📝 Medium: https://medium.com/@yusuf.ganiyu
🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
🔗 Useful Links and Resources:
✅ Docker Compose Documentation: https://docs.docker.com/compose/
✅ Apache Kafka Official Site: https://kafka.apache.org/
✅ Apache Spark Official Site: https://spark.apache.org/
✅ Confluent Docs: https://docs.confluent.io/home/overview.html
✅ S3 Documentation: https://docs.aws.amazon.com/s3/
✅ AWS IAM Documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html
✨ Tags ✨
Data Engineering, Apache Airflow, Kafka, Apache Spark, Cassandra, PostgreSQL, Zookeeper, Docker, Docker Compose, ETL Pipeline, Data Pipeline, Big Data, Streaming Data, Real-time Analytics, Kafka Connect, Spark Master, Spark Worker, Schema Registry, Control Center, Data Streaming
✨ Hashtags ✨
#confluent #DataEngineering #ApacheAirflow #Kafka #ApacheSpark #Cassandra #PostgreSQL #Docker #ETLPipeline #DataPipeline #StreamingData #RealTimeAnalytics
-
2:10:55
FreshandFit
10 hours agoAndrew Branca And Donovan Sharpe COOK Some Dumb Hoes!
175K89 -
22:26
Liberty Hangout
10 hours agoAnti-Trumper Has Complete Meltdown
18.6K40 -
2:21:24
Badlands Media
13 hours agoDevolution Power Hour Ep. 374: Declassified Intel, Trump’s Traps, and the Optics of Collapse
129K35 -
2:00:38
Inverted World Live
7 hours agoWhite House Boosts the AI Revolution | Ep. 79
44.7K13 -
10:23:56
Dr Disrespect
16 hours ago🔴LIVE - DR DISRESPECT - 10 WINS CHALLENGE - BIG ANNOUNCEMENT AT 12PM PT
159K20 -
1:35:15
Man in America
10 hours ago🚨 ALERT: Hospitals in the U.S. Are KILLING Patients… for Their Organs!
65.8K24 -
2:36:28
I_Came_With_Fire_Podcast
15 hours agoObama's Treason, Trade War: Season Infinity, and Hunter's Pipe Dream
30.7K1 -
2:52:51
TimcastIRL
8 hours agoObama Referred To DOJ For TREASON, Criminal Investigation, CIVIL WAR!! | Timcast IRL
232K138 -
9:43:32
RalliedLIVE
15 hours ago $9.38 earned10 WINS WITH THE SHOTTY BOYS
110K7 -
1:35:43
Badlands Media
18 hours agoAltered State S3 Ep. 38: Biolabs, Bitcoin & the Epstein Data Vault
68K15