Premium Only Content

Reddit Data Pipeline | AWS End to End Data Engineering
🚀 In this video, we walk you through the integration of Reddit, Airflow, Celery, Postgres, S3, AWS Glue, Athena, and Redshift to create a seamless ETL process. 📊🔍
What You Will Learn 📝:
🌐 How to extract data from Reddit using its API.
🔄 Setting up and orchestrating ETL processes with Apache Airflow and Celery.
📦 Storing efficiently with Amazon S3 using Airflow.
🧠 Leveraging AWS Glue for data cataloging and ETL jobs.
📜 Querying and transforming data with Amazon Athena.
🏢 Setting up Redshift Cluster and Best practices for loading data into Amazon Redshift for analytics.
⏰ Timestamps:
0:00 Introduction
1:27 Setting up Apache airflow with Celery Backend and Postgres
9:20 Reddit Data Pipeline with airflow
41:00 Cleaning and Transforming Reddit Data
50:00 Connecting to AWS from Airflow
1:11:17 AWS Glue data transformation
1:22:13 Querying Data with Athena
1:24:47 Setting up Redshift Data Warehouse
1:27:26 Redshift Data Warehouse Query Tool
1:29:00 Loading Data into Data Warehouse
1:32:25 Charting with Redshift Data Warehouse
🔗 Useful Links:
Reddit API Documentation: https://www.reddit.com/wiki/api/
Apache Airflow Official Site: https://airflow.apache.org/docs/
AWS Glue Documentation: https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html
💬 Let us know in the comments if you have any questions or if there's another topic you'd like us to cover next!
🌟 Don't forget to like, share, and subscribe for more data tutorials! 🌟
-
9:11
China Uncensored
9 hours agoChina Riots! Communism FAILED Yet Again.
1.56K4 -
LIVE
Robert Gouveia
2 hours ago'Missing Minute' is GONE! New IMMUNITY Demands! Obama's PLOT! Schiff in TROUBLE!
800 watching -
11:35
Tactical Advisor
2 days agoAnother Sig Trigger Fail | P320 P365 and X Macro
3.82K4 -
7:34
Michael Button
9 hours ago $0.19 earnedWhy I Left Academia to Explore Lost Civilizations
3.71K2 -
1:01:58
Sarah Westall
2 hours agoNEW STUDY RESULTS: Humans have MAC ID Chips – How Did They Get There? w/ Hazen and Mansfield
10.7K7 -
54:45
BlendrNews
2 days agoLegacy Media is Going Extinct with Sam Anthony (YourNews) | Blendr Report EP116
1342 -
30:00
BEK TV
21 hours agoCounter Culture Mom
1.08K -
2:03:53
Pop Culture Crisis
5 hours agoBillie Eilish & Sydney Sweeney Blamed For 'Whiteness', Millennials WORRIED About Gen Z | Ep. 887
30.7K12 -
2:19:34
ZiggySalvation
4 hours agoCoD HC Time
11.3K -
LIVE
Spartan
3 hours agoSpartan - Pro Halo Player for OMiT | Scrims vs C9, Maybe Ranked after
34 watching