Senior Data Engineer -Spark Architecture , PySpark and SQL $800/day
Summary
Build and enhance enterprise data platforms using PySpark, SparkSQL, and Azure for investment data processing and analytics.
Senior Data Engineer -Spark Architecture , PySpark and SQL $800/day
This initial 6mth contract will see you responsible for contributing to the overall development of our client’s enterprise data platform
Key Responsibilities for this role include :
- Build and enhance data engineering solutions
- Build and enhance lakehouses for investment data
- Implementation of a modern tech stack for data analytics, aggregation and storage of investment data, data feeds to downstream systems
Experience Requirements :
- Prior enterprise experience performing data engineering in a medallion architecture
- Extensive history of software development and data transformation experience within an enterprise environment in Australia
- Strong understanding of Spark, PySpark, SparkSQL
- Strong SQL and Python experience
- Experience with Microsoft Fabric, Azure and associated technologies building data engineering solutions within the Microsoft ecosystem
- Design and implementation of data transformation solutions based on both Parquet and SQL normalised data
- Excellent communication skills and be highly collaborative