Lead Data Engineer
Posted Updated
Summary
Lead Data Engineer at Aon's ACIA innovation hub in Singapore, designing scalable data architectures and building Databricks/Apache Spark ETL/ELT pipelines alongside actuaries, analysts and architects. Core stack: Databricks, Spark/PySpark, Python, SQL, Terraform, and AWS or Azure cloud data services.
Aon is in the business of better decisions
At Aon, we shape decisions for the better to protect and enrich the lives of people around the world.
As an organization, we are united through trust as one inclusive, diverse team, and we are passionate about helping our colleagues and clients succeed.
Aon's breadth of product offerings and global reach give us massive access to data and information, which in turn provides a unique opportunity to innovate and build and differentiating solutions for clients. But for Aon to capitalize on these opportunities, it requires a globally coordinated effort, an effort that the Aon Centre for Innovation and Analytics (ACIA) in Singapore is heavily investing towards.
What The Day Will Look Like
Collaborate with teams of experienced actuaries, data analysts, architects, data engineers and business experts to deliver projects.
Design and implement scalable data architectures that align with business objectives, including data models, meta data management, and database schemas.
Define standards, frameworks, and best practices for Spark-based ETL/ELT Development.
Drive automation and infrastructure-as-code practices (e.g., Terraform, Databricks CLI/API).
Perform data analysis to troubleshoot and resolve data-related issues.
Implement data warehouse and lake house solutions to support data and analytics initiatives.
Assist in the support and enhancements of existing data pipelines and databases and troubleshoot performance issues.
Participate in database design and modelling.
Provide technical leadership and guidance to the data engineering team in implementing architectural designs.
Work closely with cross-functional teams to understand business requirements and translate them into scalable and efficient data solutions.
Stay current with industry trends and emerging technologies to drive continuous improvement in data architecture strategies and best practices.
How This Opportunity Is Different We are seeking a highly skilled
Lead Data Engineer
with deep expertise in
Databricks
and
Apache Spark
to lead the design and implementation of data architectures that support our data-driven initiatives. This role is perfect for someone who is a great team mate, keen to learn new skills, and has a meticulous eye for data. You will work with a team of experienced actuarial analysts, architects, developers, and business resources to build databases, warehouses, and ETL/ELT pipelines to fuel modern web applications and visualizations. Skills And Experience That Will Lead To Success 8+ years of total experience working with data integration teams. 5+ years of experience in data architecture, database design, and data management 4+ years of in-depth experience developing data pipelines within an Apache Spark environment (preferably Databricks). 3+ years of active hands-on work with Databricks, demonstrating in-depth experience. Strong hands-on knowledge on Pyspark, Python and SQL and distributed computing principles. Experience in tuning spark jobs for performance and cost, addressing complex issues such as data skew, shuffle inefficiencies, and memory leaks, etc. Strong knowledge of data modelling, database technologies, data warehousing. Experience with ETL/ELT processes and tools. Knowledge of cloud platforms (Aws or Azure) and big data technologies (Hadoop, Spark, etc.) Understanding of data governance, security, and privacy frameworks. Experience with BI and analytics tools like Power BI or Tableau. Excellent communication skills, with the ability to convey complex technical concepts to both technical and non-technical stakeholders.
Nice To Haves Experience integrating AI into data engineering. Certifications in Databricks, cloud computing or database management. Experience developing data solutions using native IaaS and PaaS solutions on AWS (RDS, S3). Experience in any object-oriented programming language.
How We Support Our Colleagues In addition to our comprehensive benefits package, we encourage a diverse workforce. Plus, our agile, inclusive environment allows you to manage your wellbeing and work/life balance, ensuring you can be your best self at Aon. Furthermore, all colleagues enjoy two 'Global Wellbeing Days' each year, encouraging you to take time to focus on yourself. We offer a variety of working style solutions, but we also recognise that flexibility goes beyond just the place of work... and we are all for it. We call this Smart Working! Our continuous learning culture inspires and equips you to learn, share and grow, helping you achieve your fullest potential. As a result, at Aon, you are more connected, more relevant, and more valued. Aon values an innovative, diverse workplace where all colleagues feel empowered to be their authentic selves. Aon is proud to be an equal opportunity workplace. Aon provides equal employment opportunities to all employees and applicants for employment without regard to race, color, religion, creed, sex, sexual orientation, gender identity, national origin, age, disability, veteran, marital, domestic partner status, or other legally protected status. We welcome applications from all and provide individuals with disabilities with reasonable adjustments to participate in the job application, interview process and to perform essential job functions once onboard. If you would like to learn more about the reasonable accommodations we provide, email (HIDDEN TEXT) 2559015
How This Opportunity Is Different We are seeking a highly skilled
Lead Data Engineer
with deep expertise in
Databricks
and
Apache Spark
to lead the design and implementation of data architectures that support our data-driven initiatives. This role is perfect for someone who is a great team mate, keen to learn new skills, and has a meticulous eye for data. You will work with a team of experienced actuarial analysts, architects, developers, and business resources to build databases, warehouses, and ETL/ELT pipelines to fuel modern web applications and visualizations. Skills And Experience That Will Lead To Success 8+ years of total experience working with data integration teams. 5+ years of experience in data architecture, database design, and data management 4+ years of in-depth experience developing data pipelines within an Apache Spark environment (preferably Databricks). 3+ years of active hands-on work with Databricks, demonstrating in-depth experience. Strong hands-on knowledge on Pyspark, Python and SQL and distributed computing principles. Experience in tuning spark jobs for performance and cost, addressing complex issues such as data skew, shuffle inefficiencies, and memory leaks, etc. Strong knowledge of data modelling, database technologies, data warehousing. Experience with ETL/ELT processes and tools. Knowledge of cloud platforms (Aws or Azure) and big data technologies (Hadoop, Spark, etc.) Understanding of data governance, security, and privacy frameworks. Experience with BI and analytics tools like Power BI or Tableau. Excellent communication skills, with the ability to convey complex technical concepts to both technical and non-technical stakeholders.
Nice To Haves Experience integrating AI into data engineering. Certifications in Databricks, cloud computing or database management. Experience developing data solutions using native IaaS and PaaS solutions on AWS (RDS, S3). Experience in any object-oriented programming language.
How We Support Our Colleagues In addition to our comprehensive benefits package, we encourage a diverse workforce. Plus, our agile, inclusive environment allows you to manage your wellbeing and work/life balance, ensuring you can be your best self at Aon. Furthermore, all colleagues enjoy two 'Global Wellbeing Days' each year, encouraging you to take time to focus on yourself. We offer a variety of working style solutions, but we also recognise that flexibility goes beyond just the place of work... and we are all for it. We call this Smart Working! Our continuous learning culture inspires and equips you to learn, share and grow, helping you achieve your fullest potential. As a result, at Aon, you are more connected, more relevant, and more valued. Aon values an innovative, diverse workplace where all colleagues feel empowered to be their authentic selves. Aon is proud to be an equal opportunity workplace. Aon provides equal employment opportunities to all employees and applicants for employment without regard to race, color, religion, creed, sex, sexual orientation, gender identity, national origin, age, disability, veteran, marital, domestic partner status, or other legally protected status. We welcome applications from all and provide individuals with disabilities with reasonable adjustments to participate in the job application, interview process and to perform essential job functions once onboard. If you would like to learn more about the reasonable accommodations we provide, email (HIDDEN TEXT) 2559015