Lead Data Engineer

In Great Eastern, the Data Engineering and Business Intelligence team enable data practitioners with Data and Analytics assets. We build datamarts and visual dashboards to realize the full value of our data. We are seeking an experienced Data Engineer to join the Data Foundation team to design, build, and enhance enterprise data ingestion, transformation, and data quality capabilities. The role will focus on developing reliable, scalable, and reusable data pipelines that support analytics, reporting, regulatory, and AI use cases across the organization. The right candidate will have strong knowledge of Data Engineering and proven ability to strategize the implementation of data and analytics capabilities on a data platform (on premise and cloud) from conception through release and production. The ideal candidate will be a strong data engineer; be willing to wrangle data, optimize data systems and products, and build them from the ground up. Experience with insurance industry or Cloud technology (AWS / Google) would be an advantage.

The Lead Data Engineer will work across IT, Data Management and Governance, Visual Analytics, Data Scientists and key Business users to deliver business analytics contextual datasets and data products that enable data-driven businesses.

• Design and develop end-to-end data engineering solutions, including data ingestion, transformation, integration, and provision of trusted data assets for analytics, reporting, and business consumption.
• Adopt the best data engineering practices to design and build reliable data marts in the Hadoop ecosystem for planning, reporting, and analytics
• Build and maintain reusable enterprise data assets, canonical data models, and common data services to promote consistency, scalability, and reuse across multiple business domains.
• Work closely with business stakeholders, data scientists and data analysts to communicate data requirements, collect data, and validate quality of data products
• Collaborate with business stakeholders, data scientists and data analysts to align logic of key metrics, and to ensure code logics correctly reflect latest business definitions
• Maintain and optimize data pipelines to ensure all data is up to date with data accuracy and integrity
• Work across various stakeholders to ensure smooth production deployment of data pipeline and adherence to data governance policies.
• Proactively identify and suggest solutions to improve data engineering proces

• Bachelor’s degree in computer engineering, Computer Science, Mathematics, Software Engineering, equivalent fields or proven experience in data engineering
• Stakeholder Management – Conversant in Business terms and ability to resolve and explain data analytics issues with Business users and other concerned stakeholders
• A minimum of 10 years of experience in data engineering field. An analytics practitioner with proven experience in delivering data-driven business solutions and data-driven process augmentation
• The candidate must have demonstrated experience working with varied data infrastructure inclusive of relational databases such as SQL, Hive, Cloudera, Hadoop, Spark
• Hands-on experience with large volumes of data using Strong SQL, Spark/PySpark, Hadoop, or other big data ecosystems is preferred o Process oriented, and able to translate complex problems into logical and repeatable processes and diligently document the proposed technical solution

• Prior experience in creating, managing data models, ETL and data platforms migration is preferred

• P

  rior experience in managing codes using GIT Version control ,

  Understanding of banking, insurance and financial services is preferred

• Minimum 10 years of working experience in SQL / Hive QL / Spark
• Minimum 10 years of programming experience in preferred languages Python, PySpark
• Minimum 10 years of experience in data engineering work on Big Data Technology (Hortonworks / Cloudera)
• Knowledge of data quality principles and experience implementing data quality controls, validation frameworks, reconciliation processes, and monitoring mechanisms within data pipelines.
• Understanding of metadata management, data lineage, and data governance concepts.
• Experience working with enterprise data models, reusable data assets, and data integration frameworks is preferred.
• Able to work with specialists in different disciplines to formulate data solutions
• Able to work collaboratively as part of a team.
• High level of integrity, takes accountability of work and good attitude over teamwork.
• Takes initiative to improve current state of things and adaptable to embrace new changes.