Senior Data & Analytics Engineer - Databricks
Summary
A hands-on senior data engineer role building and maintaining end-to-end Databricks (PySpark/SQL) pipelines using Medallion architecture, Unity Catalog and Delta Live Tables, with dimensional data modelling for sales, operations, marketing and finance reporting, BI semantic layers, Spark performance tuning and data security for a business in Western Sydney, NSW.
Seeking a Senior Data & Analytics Engineer with recent Databricks for this rapidly expanding and successful business in Western Sydney!
Looking for a high-performing hands on Data Engineer to design and implement robust data pipelines using Databricks with a focus on Data Modelling for sales, operations marketing and financial workstreams.
Key responsibilities
- Delivering solutions: Defining the problem statements, conducting the business analysis and building requirements, design and delivering solutions
- Pipeline Architecture: Design, build, and maintain scalable end-to-end data pipelines in Databricks (PySpark/SQL). Includes orchestrate Medallion-architecture pipelines in Databricks (Unity Catalog, Delta Live Tables).
- Advanced Data Modelling: Develop star/snowflake schemas and vault models that unify disparate data from various ERP sources.
- Domain Expertise: Build complex logic for sales, operations and financial reporting, including and analytics (pipeline health, churn, conversion).
- BI Delivery: Develop and optimise high-performance semantic layers in to ensure "one version of the truth."
- Training users: provide training to users
- Performance Tuning: Optimise large-scale Spark jobs to ensure lightning-fast reporting for stakeholders. Includes optimisation, indexing strategies, and partition management to reduce latency in BI reports.
- Security & Governance: Implement Row-Level Security (RLS) and Object-Level Security (OLS) across Databricks and SQL environments to ensure sensitive financial data is restricted.
Success metrics
- Business value: User satisfaction and utilisation of the data services by our users in running the day-to-day operations
- Data Reliability: Zero "Data Mismatch" incidents reported by the Finance team during month-end closing.
- Operational Excellence: Implementation of automated alerts for pipeline failures and database performance bottlenecks
- Cost Efficiency: Reduction in monthly Databricks/Cloud spend through optimised SQL and cluster management.
- Reporting Speed: load times for the primary finance & Sales dashboards.
Required skills
- Core Tech: Solid Data Engineering experience with a focus on Databricks (Medallion architecture)
- Data Modelling: Proven expertise in dimensional modelling and translating complex business processes into logical data structures.
- BI Tools: Expert-level proficiency in a major BI tool for complex calculated fields.
- Programming: Strong proficiency in Python (PySpark) and SQL.
What's in it for you?
- A friendly, professional environment that has been recognised as a 'Great Place to Work' for five years consecutively
- A greenfield opportunity to make your mark on a true Business & Technology Transformation
- Outstanding career development and learning
- Work on multiple CRM and ERP platforms including SAP S/4HANA Public Cloud