Premium Only Content

Reddit Data Pipeline | AWS End to End Data Engineering
🚀 In this video, we walk you through the integration of Reddit, Airflow, Celery, Postgres, S3, AWS Glue, Athena, and Redshift to create a seamless ETL process. 📊🔍
What You Will Learn 📝:
🌐 How to extract data from Reddit using its API.
🔄 Setting up and orchestrating ETL processes with Apache Airflow and Celery.
📦 Storing efficiently with Amazon S3 using Airflow.
🧠 Leveraging AWS Glue for data cataloging and ETL jobs.
📜 Querying and transforming data with Amazon Athena.
🏢 Setting up Redshift Cluster and Best practices for loading data into Amazon Redshift for analytics.
⏰ Timestamps:
0:00 Introduction
1:27 Setting up Apache airflow with Celery Backend and Postgres
9:20 Reddit Data Pipeline with airflow
41:00 Cleaning and Transforming Reddit Data
50:00 Connecting to AWS from Airflow
1:11:17 AWS Glue data transformation
1:22:13 Querying Data with Athena
1:24:47 Setting up Redshift Data Warehouse
1:27:26 Redshift Data Warehouse Query Tool
1:29:00 Loading Data into Data Warehouse
1:32:25 Charting with Redshift Data Warehouse
🔗 Useful Links:
Reddit API Documentation: https://www.reddit.com/wiki/api/
Apache Airflow Official Site: https://airflow.apache.org/docs/
AWS Glue Documentation: https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html
💬 Let us know in the comments if you have any questions or if there's another topic you'd like us to cover next!
🌟 Don't forget to like, share, and subscribe for more data tutorials! 🌟
-
15:30
BlackBeltBarrister
17 hours ago $11.30 earnedThey will screw us all with this!!
64.2K31 -
2:04:22
TimcastIRL
13 hours agoDan Bongino AND Kash Patel THREATEN TO QUIT FBI Over Botched Epstein Release | Timcast IRL
308K300 -
5:10:31
Turning Point USA
17 hours agoSAS IS LIVE! Pete Hegseth, Don Jr., Tucker Carlson, Charlie Kirk and more... in Tampa, FL!
915K100 -
1:54:35
Glenn Greenwald
19 hours agoUN Gaza Investigator Francesca Albanese on US Sanctions Against Her; Plus: Glenn Takes Your Questions on Trump's Pressure on Brazil, Sam Harris, Bill Ackman, and More | SYSTEM UPDATE #485
108K61 -
58:37
BonginoReport
23 hours agoElon’s Petty Move Threatens GOP’s Bright Future - Nightly Scroll w/ Hayley Caronia (Ep.88)
123K136 -
24:53
Preston Stewart
18 hours ago $4.44 earnedTrump Arms Ukraine
89.4K60 -
1:00:00
BEK TV
1 day agoDRONES IN AMERICA: DJI'S MARKET DOMINANCE, FUTURE USES, AND A REAL LOOK AT THE MEASLES OUTBREAK
56.7K7 -
8:22:22
Dr Disrespect
22 hours ago🔴LIVE - DR DISRESPECT - INSANE WARZONE CHALLENGES! DOMINATING 2025 HYPE
206K8 -
12:55:42
LFA TV
1 day agoLFA TV ALL DAY STREAM - FRIDAY 7/11/25
277K44 -
1:13:26
Kim Iversen
18 hours agoDershowitz Says He Knows FOR A FACT The Names On The Epstein List
90.5K201