GCP Data Engineer (Databricks)
Posted Updated
Description
Role Overview
We are looking for a Senior Data Engineer with strong hands-on experience in building and optimizing scalable data pipelines on Google Cloud Platform (GCP) and Databricks. This role requires deep technical expertise in data engineering, pipeline development, and cloud-native solutions, with the ability to work closely with cross-functional teams to deliver high-quality data systems.
We are looking for a Senior Data Engineer with strong hands-on experience in building and optimizing scalable data pipelines on Google Cloud Platform (GCP) and Databricks. This role requires deep technical expertise in data engineering, pipeline development, and cloud-native solutions, with the ability to work closely with cross-functional teams to deliver high-quality data systems.
Key Responsibilities
- Design, build, and optimize scalable batch and real-time data pipelines on GCP and Databricks.
- Develop and maintain ETL/ELT workflows for diverse data sources.
- Build robust data ingestion frameworks and support data migration initiatives.
- Develop and optimize Spark/PySpark workloads in Databricks, including notebooks, jobs, and Delta Lake tables.
- Manage workflow orchestration, scheduling, and pipeline dependencies (Cloud Composer, Databricks Workflows).
- Ensure efficient and reliable data processing across environments.
- Optimize query performance and data processing jobs, including Databricks cluster and job tuning.
- Implement data validation, quality, and reconciliation frameworks.
- Monitor pipelines to ensure availability, reliability, and performance.
- Collaborate with cross-functional teams to translate business requirements into solutions.
- Contribute to data architecture and design discussions.
- Follow best practices for version control, CI/CD, and deployments.
- Ensure compliance with security, governance, and data standards, including access control and cataloging in Databricks.
- Support production systems through troubleshooting and continuous improvement.
Must-Have Skills
- 6-8 years of experience in data engineering.
- Strong hands-on experience with GCP (BigQuery, Dataflow, Pub/Sub, Cloud Storage).
- Hands-on experience with Databricks, including Delta Lake, Databricks Workflows/Jobs, and notebook-based development.
- Experience with Spark / PySpark and SQL optimization.
- Hands-on with Cloud Composer (Airflow).
- Strong understanding of data modeling and pipeline design.
- Experience with batch + streaming data processing.
- Familiarity with CI/CD and Git-based workflows.
- Strong problem-solving and analytical skills
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.