Senior Data Engineer (or Assistant Manager) - Databricks

Responsibilities

• Design and maintain scalable ETL/ELT pipelines in Azure Databricks to process structured, semi-structured, and unstructured insurance data from diverse sources.

• Collaborate with architects, modellers, analysts, and stakeholders to deliver tailored data assets for analytics, operations, and regulatory reporting.

• Develop and optimize batch and streaming data solutions using Azure Data Factory, Data Lake Storage Gen2, Azure Event Hubs, and Kafka.

• Implement robust data validation, cleansing, and quality controls to ensure reliability for critical insurance use cases.

• Integrate Informatica tools to support governance, metadata management, lineage tracking, and cataloguing.

• Enforce data security best practices including RBAC, managed identities, encryption, and compliance with PDPO, GDPR, and other regulations.

• Automate deployment workflows using GitHub Actions for efficient, repeatable, and auditable data operations.

• Troubleshoot pipeline issues, conduct root cause analysis, and proactively resolve data quality and performance challenges.

• Maintain detailed technical documentation for data pipelines, transformation logic, and operational procedures.

• Apply domain expertise in Hong Kong Life and General Insurance to ensure solutions meet local business and regulatory standards.

Requirements

• Bachelor’s degree in Computer Science, IT, or a related field.

• 5+ years of experience with Azure cloud platforms, focused on Databricks for insurance data workloads.

• Strong skills in provisioning and managing Azure Databricks clusters and workspaces for varied data types.

• Experience integrating Azure Data Factory and Data Lake Storage Gen2 with Databricks for seamless data flows.

• Proficient in Terraform for infrastructure-as-code deployment of Azure and Databricks services.

• Hands-on experience with Informatica for metadata management, cataloguing, and governance.

• Deep understanding of Azure security (RBAC, NSG, Key Vault), monitoring, and cost optimization in regulated environments.

• Skilled in CI/CD automation using GitHub Actions for platform and pipeline deployments.

• Proven ability to troubleshoot, optimize, and support mission-critical data workloads in insurance.

• Strong documentation and communication skills to support cross-functional teams and stakeholders.

**This position is offered through a strategic partnership between Don Nelson Technology Limited and NP Solution Limited.