Senior Data Engineer

We are looking for a Senior Data Engineer with strong experience in AWS, Databricks, Python, PySpark and SQL to design, develop and maintain scalable data pipelines and data platforms.

• Design and develop batch and streaming data pipelines using AWS and Databricks.
• Build and maintain data lakehouse solutions using Delta Lake.
• Develop data ingestion, transformation and data modelling processes.
• Implement data quality checks, validation and data governance.
• Automate workflows using Databricks and AWS services.
• Monitor and troubleshoot data pipelines and production issues.
• Optimise data processing performance and cloud costs.
• Implement data security, access controls and encryption.
• Review code, follow engineering standards and mentor team members.
• Maintain technical documentation and operational procedures.

• Strong experience in Python, PySpark and SQL.
• Hands-on experience with Databricks, Apache Spark and Delta Lake.
• AWS data services such as S3, Glue, IAM, Lambda and Redshift.
• Experience in data lake/lakehouse architecture and data modelling.
• Knowledge of incremental data processing and Change Data Capture (CDC).
• Experience with data quality, workflow orchestration and monitoring.
• Knowledge of Git, CI/CD and Terraform or equivalent IaC tools.
• Experience in performance tuning, troubleshooting and cloud security.
• Technical leadership, code reviews or mentoring experience.

• Databricks Unity Catalog, Lakeflow and Structured Streaming.
• Apache Airflow, dbt, Kafka or AWS Kinesis.
• Experience with ML/AI data pipelines.
• Relevant AWS or Databricks certifications.

Bachelor?s degree in Computer Science, Information Technology, Data Engineering or a related field.