Premium Only Content

Robust Data Pipelines with Apache Spark, DBT and Azure | End-to-End Data Engineering Project
As decided by the community, here is a teaser for the Apache Spark, Databricks, DBT and Cloud Provider project.
Timestamp:
0:00 Introduction
0:49 System Architecture
3:01 Creating resource groups on Azure
5:02 Setting up the medallion architecture storage account
8:46 Setting up Azure Data Factory
10:18 Azure Key Vault setup for secrets
14:19 Azure database with automatic data population
25:32 Azure Data Factory pipeline orchestration
47:00 Setting up Databricks
49:50 Azure Databricks Secret Scope and Key Vault
54:33 Verifying Databricks - Key Vault - Secret Scope Integration
1:06:00 Azure Data Factory - Databricks Integration
1:21:19 DBT Setup
1:24:15 DBT Configuration with Azure Databricks
1:32:12 DBT Snapshots with Azure Databricks and ADLS Gen2
1:45:06 DBT Datamarts with Azure Databricks and ADLS Gen2
1:55:00 DBT Documentation
1:58:58 Outro
If you find our content valuable, support us by joining our channel membership, where you'll get exclusive access to behind-the-scenes content, Q&A sessions, and much more!
https://www.youtube.com/channel/UCAEOtPgh29aXEt31O17Wfjg/join
💬 Join the Conversation:
We love hearing from you! Share your thoughts, questions, or experiences related to data engineering or this project in the comments below. Don't forget to like, subscribe, and hit the bell icon to stay updated with our latest content.
Tags:
Big Data, Data Engineering, Apache Spark, Databricks, DBT, Azure, Cloud Computing, Data Analytics, ETL, Data Warehouse, Technology, Analytics, Machine Learning, Data Science
Hashtags:
#BigData, #DataEngineering, #ApacheSpark, #Databricks, #DBT, #Azure, #CloudComputing, #DataAnalytics, #ETL, #DataWarehouse, #TechTalk, #MachineLearning, #DataScience, #BigDataAnalytics
🙏 Thank You for Watching!
Remember to subscribe and hit the bell icon for notifications. Stay curious and keep exploring the fascinating world of data engineering!
-
LIVE
Barry Cunningham
2 hours agoPRESIDENT TRUMP TOURS JEROME POWELL SCENE OF THE CRIME AT THE FEDERAL RESERVE
2,962 watching -
LIVE
Sarah Westall
1 hour agoBlackmail, Power & Corruption: The Currency of the Empire - Epstein & more w/ Joachim Hagopian
214 watching -
45:56
The White House
2 hours agoPresident Trump Visits the Federal Reserve
3.24K12 -
36:14
Kimberly Guilfoyle
3 hours agoRussia Hoax Reality Check, Live! | Ep240
4.57K5 -
11:02
Preston Stewart
6 hours agoThailand–Cambodia Clash Erupts
372 -
LIVE
LFA TV
20 hours agoLFA TV ALL DAY STREAM - THURSDAY 7/24/25
1,355 watching -
1:07:35
vivafrei
3 hours agoWall Street Journal DOUBLING DOWN on Epstein! Lawyer in Canada DEBANKED! AND MORE!
83.9K24 -
4:42:20
Donut Operator
6 hours agoI'M BACK/ CRIME/ GAMEBOY CAMERA CHAD
88.6K4 -
9:51
Warren Smith - Secret Scholar Society
5 days agoThe Moment Tim Pool Discovered Why Andrew Wilson is so Unstoppable
27.7K26 -
28:23
Film Threat
5 hours agoLIVE FROM SAN DIEGO COMIC-CON! (Thursday) | Film Threat Live
12.9K