Robust Data Pipelines with Apache Spark, DBT and Azure | End-to-End Data Engineering Project
As decided by the community, here is a teaser for the Apache Spark, Databricks, DBT and Cloud Provider project.
Timestamp:
0:00 Introduction
0:49 System Architecture
3:01 Creating resource groups on Azure
5:02 Setting up the medallion architecture storage account
8:46 Setting up Azure Data Factory
10:18 Azure Key Vault setup for secrets
14:19 Azure database with automatic data population
25:32 Azure Data Factory pipeline orchestration
47:00 Setting up Databricks
49:50 Azure Databricks Secret Scope and Key Vault
54:33 Verifying Databricks - Key Vault - Secret Scope Integration
1:06:00 Azure Data Factory - Databricks Integration
1:21:19 DBT Setup
1:24:15 DBT Configuration with Azure Databricks
1:32:12 DBT Snapshots with Azure Databricks and ADLS Gen2
1:45:06 DBT Datamarts with Azure Databricks and ADLS Gen2
1:55:00 DBT Documentation
1:58:58 Outro
If you find our content valuable, support us by joining our channel membership, where you'll get exclusive access to behind-the-scenes content, Q&A sessions, and much more!
https://www.youtube.com/channel/UCAEOtPgh29aXEt31O17Wfjg/join
💬 Join the Conversation:
We love hearing from you! Share your thoughts, questions, or experiences related to data engineering or this project in the comments below. Don't forget to like, subscribe, and hit the bell icon to stay updated with our latest content.
Tags:
Big Data, Data Engineering, Apache Spark, Databricks, DBT, Azure, Cloud Computing, Data Analytics, ETL, Data Warehouse, Technology, Analytics, Machine Learning, Data Science
Hashtags:
#BigData, #DataEngineering, #ApacheSpark, #Databricks, #DBT, #Azure, #CloudComputing, #DataAnalytics, #ETL, #DataWarehouse, #TechTalk, #MachineLearning, #DataScience, #BigDataAnalytics
🙏 Thank You for Watching!
Remember to subscribe and hit the bell icon for notifications. Stay curious and keep exploring the fascinating world of data engineering!
-
1:27:47
CodeWithYu
8 months agoRealtime Data Streaming | End To End Data Engineering Project
30 -
58:05
CodeWithYu
7 months agoApache Flink For Sales Analytics - End to End Data Engineering
7 -
1:18:21
Los Angeles Data Platform
3 years agoMarch 2021 - Spark and Databricks in Azure Synapse by Ginger Grant (@desertislesql)
19 -
3:46
AHMEDKAZEKA97
1 year agoOVERVIEW System Design for Big Data Pipelines - UDEMY COURSE
42 -
3:19
hayksimonyanrumble
8 months agoSystem Design Concepts: Architecture of Production Web Apps
9 -
7:49
EkasCloud Online Courses
1 year ago#Azure Understanding Pipelines|English|Ekascloud
3 -
0:08
danilosousa
1 year agoApache Spark
3 -
1:22:21
Jake's DevLab
6 months agoLive Coding a Full-Stack Service Request System with AWS & React - From DynamoDB to SES Integration!
1322 -
15:34
ideadudes
3 years agoAzure Monitor - Service Health and Performance Tools
50 -
11:21
EkasCloud Online Courses
1 year ago#Azure Devops Pipelines | English| Ekascloud
2