Senior Azure Databricks Engineer - Spark & Data Platform
Posted Updated
Description
Digifloat in Islamabad, Pakistan, is seeking a Senior Databricks Engineer with 7-8 years of experience to design, build, and optimize scalable data platforms. You will own end-to-end data pipelines and data lake architectures, collaborating with stakeholders to deliver reliable, governed data solutions.
You will lead ETL/ELT development, optimize Spark workloads, and mentor junior engineers, while aligning with security, cost, and performance objectives in a modern data environment.
Design, develop, and maintain scalable data pipelines using Azure Databricks and Apache Spark. Build and optimize ETL/ELT processes for large-scale data integration and transformation. Develop and manage Delta Lake architectures to ensure data reliability, governance, and performance. Implement and maintain data ingestion frameworks from various structured and unstructured data sources. Collaborate with data architects, analysts, and business teams to translate requirements into technical solutions. Monitor, troubleshoot, and optimize Databricks workloads for performance and cost efficiency. Ensure adherence to data governance, security, and compliance standards. Mentor junior engineers and contribute to technical design discussions and architecture reviews. Participate in code reviews and drive engineering best practices across the team. Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related field. 7-8 years of overall experience in Data Engineering. Minimum 4+ years of hands-on experience with Azure Databricks. Strong expertise in Apache Spark (PySpark and/or Scala). Advanced SQL development and performance tuning skills. Experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), and Azure Synapse Analytics. Strong understanding of Delta Lake architecture and Lakehouse concepts. Experience building robust ETL/ELT pipelines in enterprise environments. Knowledge of data modeling, warehousing, and dimensional modeling concepts. Experience with Git, DevOps practices, and CI/CD pipelines. Strong troubleshooting, analytical, and problem-solving skills.Skills
Sign in to read and join the discussion.