Data Engineer
Posted Updated
Description
XPT Software is seeking a Senior Data Engineer in Sydney, Australia, to design, build, and support scalable data pipelines and lakehouse solutions using Azure Databricks and the Microsoft Azure data ecosystem.
You will design batch and streaming data pipelines, implement lakehouse patterns, integrate with Azure Data Lake Gen2, Data Factory, Key Vault, and DevOps, and lead production-grade Databricks notebooks, jobs, and CI/CD workflows.
Design, build and optimize scalable batch and streaming pipelines. Develop ingestion, transformation, validation, and publishing frameworks. Implement lakehouse patterns across data zones. Integrate Databricks with Azure services and manage notebooks and workflows. Create data quality checks, lineage, and monitoring; optimize performance. Lead code reviews, set standards, and mentor other engineers. 8+ years in data engineering or large-scale data platforms. Hands-on with Azure Databricks, PySpark, Python and SQL. Experience with Delta Lake, lakehouse patterns, and data lake design. CI/CD using Azure DevOps, Git and automated testing. Knowledge of data governance, security and metadata concepts.Skills
Sign in to read and join the discussion.