Overview
Set up Databricks on Google Cloud (GCP) including workspace, Unity Catalog, clusters, and GCS integration., Implement the Medallion Architecture (Bronze, Silver, Gold) using PySpark and Databricks Auto Loader for structured streaming., Build end-to-end ETL pipelines that load raw data, perform transformations, and generate business-ready gold tables for analytics., Automate workflows and orchestrate pipelines in Databricks with parameterized jobs and file arrival triggers., Integrate GitHub with Databricks and manage code promotion across Dev, UAT, and PRD environments using pull requests., Analyze real-world road traffic data to derive insights such as busiest regions, EV adoption trends, and yearly traffic volume patterns.
Aspiring Data Engineers who want to gain practical, hands-on experience to crack interviews, Cloud Professionals (GCP/Azure/AWS) looking to expand their skills into Databricks and Medallion architecture., Students & Beginners in Data Engineering who want a guided, real-world project to add to their portfolio.
Basic knowledge of SQL PySpark and data concepts, Enthusiasm to learn hands-on with a real-world project – no prior Databricks experience required!
Are you looking to master Databricks on Google Cloud (GCP) with a real-world, end-to-end project? This course is designed to give you hands-on experience with one of the most in-demand skills in data engineering: building scalable data pipelines using Databricks, PySpark, and Medallion Architecture.
In this project, we take on the role of a government transport agency analyzing road traffic data. You will learn how to manage road infrastructure datasets, process traffic counts from sensors, and generate insights such as the busiest regions, EV adoption trends, and year-over-year traffic volume.
We will start by setting up Databricks on GCP, creating buckets, external locations, and Unity Catalog, and then move step by step through Bronze, Silver, and Gold layers using Auto Loader and Structured Streaming. You’ll gain real-world exposure to data ingestion, transformation, and aggregation pipelines.
The course also goes beyond development by covering workflow orchestration, GitHub integration, and CI/CD practices. You will learn how to set up Dev, UAT, and Production environments, manage code using Git branches, and promote pipelines using pull requests – just like in real industry projects.
By the end of this course, you will not only have built a portfolio-ready project, but also be equipped with the practical knowledge and interview-ready concepts to crack Data Engineering roles involving Databricks and GCP.
Whether you’re a beginner or a working professional, this project-based course ensures you learn by doing – and walk away with confidence in both technical skills and real-world applications.
Saidhul Shaik
I am a seasoned Cloud Data Engineer with expertise in GCP, Azure, and AWS, specializing in data engineering, analytics, and DevOps. With years of hands-on experience in building scalable data pipelines, optimizing Apache Spark, and managing cloud migrations, I am passionate about helping learners bridge the gap between theory and real-world applications. As the founder of Skill Vane Software Institute, I mentor aspiring data engineers, guiding them through industry-standard best practices, hands-on projects, and interview preparation. My goal is to empower professionals with the skills needed to thrive in cloud and big data ecosystems.