Data Engineer

Key Responsibilities

• Design, develop and implement scalable data and platform solutions across AWS cloud and hybrid/on-premises environments.

• Build, maintain and optimise production-grade data pipelines, platform services and automation frameworks using Python, PySpark, SQL and cloud-native technologies.

• Support and enhance enterprise data platforms, including data lakes, lakehouse architectures, metadata management, search platforms and AI services.

• Develop and maintain CI/CD pipelines, automated testing frameworks and deployment processes to ensure reliable and efficient software delivery.

• Implement, integrate and support technologies such as OpenSearch, Amazon Bedrock, AgentCore, MCP and related platform services.

• Monitor and improve platform reliability, security, performance, observability and cost efficiency.

• Troubleshoot and resolve production issues, perform root-cause analysis and ensure platform stability and operational excellence.

• Collaborate with cross-functional teams to drive platform modernisation, automation and continuous improvement initiatives.

Mandatory Requirements

• 3–5+ years of hands‑on experience building and supporting production‑grade data pipelines using SQL, Python and cloud technologies.

• Practical experience working with AWS and/or hybrid cloud environments.

• Strong Linux/Unix administration and scripting skills, with experience supporting production systems and troubleshooting operational issues.

• Strong understanding of ETL/ELT processes, data integration, data transformation and data quality management.

• Experience with distributed data processing frameworks and Data Lake/Lakehouse architectures.

• Proficiency in Git, CI/CD pipelines, DevOps practices and production deployment processes.

Technical Requirements

Data Engineering & Processing

• Python, PySpark, SQL, Apache Spark and Airflow.

• ETL/ELT development, data modelling and distributed data processing.

• Data Lake/Lakehouse architectures and table formats such as Apache Iceberg.

AWS & Cloud Platform

• AWS Glue, EMR, S3, Redshift and Lambda.

• Kafka/MSK, DBT, Hive, Trino and Presto.

• OpenSearch, Amazon Bedrock, AgentCore and Model Context Protocol (MCP).

Platform Engineering & Operations

• Linux/Unix administration, shell scripting and production support.

• Git, CI/CD, DevOps practices and Infrastructure as Code (IaC).

• Monitoring, troubleshooting, performance tuning and platform reliability.

Preferred Qualifications

• Experience with enterprise Data Lake and Lakehouse platforms.

• Knowledge of streaming and event‑driven architectures.

• Hands‑on experience with OpenSearch administration, performance optimisation and tuning.

• Familiarity with monitoring and observability tools such as Datadog, Amazon CloudWatch, Grafana or Prometheus.

• Understanding of data governance, security, access controls and Personally Identifiable Information (PII) protection.

• Experience in insurance, banking or financial services environments.