Premium Only Content

Realtime Streaming with Unstructured Data Engineering | Get Hired as an Experienced Data Engineer
In this video you will be building a realtime streaming pipeline for unstructured data with different data types (TEXT, IMAGE, VIDEO, CSV, JSON, PDF) with over 600+ different datasets.
MORE DATA ENGINEERING VIDEOS AVAILABLE on datamasterylab.com
Like this video?
- Buy me a coffee: https://www.buymeacoffee.com/yusuf.ganiyu
- Support the channel: https://www.youtube.com/@codewithyu/join
Timestamps:
0:00 Introduction
1:50 System Architecture Overview
4:08 System Architecture Design
13:22 Setting up Spark Streaming for Unstructured Data
21:46 Handling multiple unstructured data types
24:31 Creating data schema
30:35 Creating custom user define functions for data extraction
51:14 Parsing and extracting text data
1:40:30 Structuring the results into a dataframe
1:46:15 Reading JSON structured files into the streams
1:49:47 Joining Structured and Unstructured Data Streams
1:52:50 Writing Data to AWS S3 Bucket
2:04:20 Creating AWS Glue Crawler for the data
2:08:25 Verifying the crawler results on Athena
2:11:36 Deploying Spark Streams to Spark Clusters
2:26:31 Verification of Results
2:29:40 Outro
👦🏻 My Linkedin: https://www.linkedin.com/in/yusuf-ganiyu-b90140107/
🚀 X(Twitter): https://x.com/YusufOGaniyu
📝 Medium: https://medium.com/@yusuf.ganiyu
🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
🔗 Useful Links and Resources:
✅ Source Code and Datasets: https://www.buymeacoffee.com/yusuf.ganiyu/source-code-real-time-streaming-pipelines-unstructured-data
✅ Docker Compose Documentation: https://docs.docker.com/compose/
✅ Apache Spark Official Site: https://spark.apache.org/
✅ Confluent Docs: https://docs.confluent.io/home/overview.html
✅ S3 Documentation: https://docs.aws.amazon.com/s3/
✅ AWS IAM Documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html
✨ Tags ✨
Data Engineering, Apache Spark, Unstructured Data, Docker, Docker Compose, ETL Pipeline, Data Pipeline, Big Data, Streaming Data, Real-time Analytics, Kafka Connect, Spark Master, Spark Worker, Schema Registry, Control Center, Data Streaming
✨ Hashtags ✨
#DataEngineering #ApacheSpark #unstructureddata #Docker #ETLPipeline #DataPipeline #StreamingData #RealTimeAnalytics
-
19:18
DeVory Darkins
4 hours ago $16.19 earnedDemocrat non-profit SCAM EXPOSED as Trump OBLITERATES Newsom
23.4K50 -
23:58
Stephen Gardner
6 hours ago🔥Obama THROWS Adam Schiff under the bus to obstruct Trump!
17.5K99 -
38:44
The Why Files
4 days agoProject Ancient Arrow | The NSA's Secret War Against Our Future
52.1K72 -
2:36:06
Barry Cunningham
8 hours agoPRESIDENT TRUMP IS TRULY USHERING IN THE GOLDEN AGE OF AMERICA! CAN YOU FEEL IT?
105K44 -
3:47:25
SynthTrax & DJ Cheezus Livestreams
3 days agoFriday Night Synthwave 80s 90s Electronica and more DJ MIX Livestream 2K Celebration SPECIAL EDITION 530pm PST / 830pm EST
46.8K5 -
2:21:54
VapinGamers
4 hours ago $2.97 earnedDestiny 2 - Edge of Fate Legendary Run Part 3 - !rumbot !music
24.9K -
2:04:25
TimcastIRL
7 hours agoTrump DOJ Gives Ghislaine Maxwell Limited IMMUNITY As She Rats On 100+ People | Timcast IRL
227K165 -
1:09:09
Omar Elattar
9 hours agoThe Brain Experts: Your Brain Can Rewire Itself At Any Age & Here's How!
22.5K4 -
4:30:37
IcyFPS
5 hours agoLIVE - Wuchang Fallen Feathers x Borderlands w/ pope!
29.4K2 -
29:24
Afshin Rattansi's Going Underground
19 hours agoWas Epstein a Mossad Agent? Will Obama go to Prison? (Afshin Rattansi vs Alan Dershowitz)
30.8K23