Remote Data Engineer — Databricks & Spark Pipelines
Posted Updated
Description
Capgemini Engineering is seeking a Data Engineer to design, develop, and maintain scalable ETL/ELT data pipelines using Python and Databricks, joining a global team of engineers to help organizations leverage data for impactful results.
You will work with Spark, PySpark, Delta Lake, and Lakehouse architectures, integrating data from databases, APIs, files, and cloud storage while ensuring data quality and reliability in a remote-friendly setup.
Design, develop, and maintain scalable ETL/ELT data pipelines using Python and Databricks. Develop data processing and transformation solutions using Spark and PySpark. Build, optimize, and maintain Databricks notebooks, workflows, and jobs. Integrate data from multiple sources, including databases, APIs, files, and cloud storage, with cleansing, enrichment, and validation activities. Optimize data pipelines and Spark workloads for performance, scalability, reliability, monitoring, logging, and troubleshooting. Collaborate with analysts, developers, architects, and business teams while participating in code reviews, testing, CI/CD, and production support activities. Strong hands-on experience with Python, Databricks, Apache Spark, and PySpark. Experience building and supporting production-grade ETL/ELT data pipelines. Strong SQL skills and experience with data modeling, data transformation, and data quality practices. Experience working with Delta Lake and Lakehouse architectures. Experience integrating data from databases, APIs, files, and cloud storage, with understanding of Git, CI/CD, testing, and engineering best practices.Skills
Sign in to read and join the discussion.