Senior Data Engineer, Databricks
Summary
The Senior Data Engineer will design and optimize scalable data pipelines using Azure Databricks, Apache Spark, and Azure Data Factory. The role focuses on building data solutions for the financial sector while ensuring strict adherence to data governance and security standards.
- Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark
- Build and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichment
- Collaborate with Data Scientists to understand data requirements and deliver clean, structured datasets
- Implement data quality checks, validation rules, and monitoring mechanisms
- Integrate structured, semi-structured, and unstructured data sources
- Ensure data security, governance, and compliance with banking regulations
- Optimize Spark jobs for performance and cost-efficiency in Azure
- Participate in code reviews, design discussions, and agile ceremonies
- Document data pipelines, workflows, and technical decisions
Requirements
- At least 2+ years of hands-on experience in Azure Databricks
- Strong proficiency in Apache Spark, especially PySpark
- Experience with Azure Data Factory, Azure Synapse, Azure Blob Storage, and Delta Lake
- Solid understanding of ETL/ELT processes, data modeling, and data warehousing concepts
- Proficiency in Python and/or Scala for data processing
- Experience with CI/CD pipelines using Azure DevOps, Git, and Terraform preferred
- Familiarity with SQL, NoSQL, and data lake architectures
- Knowledge of data governance, security, and compliance in financial services
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder management abilities
- Ability to work collaboratively in cross-functional teams
- Agile mindset and experience in Scrum/Agile environments
- Bachelor’s or Master’s degree in Computer Science, Information Systems, Engineering, or a related field preferred
- Databricks Certified Data Engineer Associate/Professional certification is a plus
- Experience in banking or financial services is highly desirable
- Australian citizenship required
- Must be eligible to obtain security clearance
Core Competencies
Demonstrates expertise in designing and optimizing data ingestion pipelines using Azure Data Factory and Databricks, with a strong focus on data governance and compliance in financial services. Proficient in data processing with PySpark and Python, ensuring high-quality data delivery and collaboration with cross-functional teams.
Highest-signal resume keywords
- Azure Databricks
- Apache Spark
- Data Governance
- ETL/ELT Processes
- CI/CD Pipelines
ATS Optimization Keywords
Hard Skills
- Data Ingestion Pipelines
- PySpark
- Azure Data Factory
- SQL
- NoSQL
- Data Modeling
- Data Warehousing
- Python
- Scala
- Terraform
Soft Skills
- Analytical Skills
- Problem-Solving Skills
- Communication Skills
- Stakeholder Management
- Collaboration
Certifications & Qualifications
- Databricks Certified Data Engineer Associate
- Databricks Certified Data Engineer Professional
Industry Keywords
- Data Governance
- Data Security
- Compliance
- Banking Regulations
- Financial Services
Tools & Technologies
- Azure Synapse
- Azure Blob Storage
- Delta Lake
- Azure DevOps
- Git