Data Engineer, AI Integration Specialist
Posted Updated
Description
About the role
We are looking for a
Data Enginee r to design, build, and support reliable data ingestion and transformation pipelines on
AWS
and
Databricks . The role will work with architects, engineers, QA, and business teams to deliver data pipelines, curated datasets, data-quality controls, and production-ready operational processes. Key responsibilities
Design and build batch and/or streaming data pipelines
using
Databricks
and
AWS
services. Develop data processing across
Bronze, Silver, and Gold layers
using
SQL, Python/PySpark, and Delta Lake . Implement ingestion from APIs, files, databases, object storage, or other enterprise data sources. Develop data transformations, dimensional or curated datasets, and reusable data-engineering components. Implement data validation, reconciliation, error handling, logging, rerun/recovery, and monitoring. Optimise Spark jobs, clusters, partitioning, and storage
for performance and cost. Support CI/CD, environment promotion, SIT/UAT, production deployment, documentation, and operational handover. Collaborate with platform, application, BI, and data-governance teams. About you
3+ years of experience
in data engineering or enterprise data-platform delivery. Hands-on experience with Databricks, Apache Spark, PySpark, and SQL. Experience with Delta Lake
and layered data architecture such as
Bronze/Silver/Gold . Experience building production data pipelines
with proper logging, error handling, and recovery. Experience with AWS services
such as
S3, IAM, Glue, Lambda, Kinesis , or related services. Good understanding of data modelling, data quality, and performance optimisation. Experience with Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, or Structured Streaming. Experience with APIs, XML/JSON, event or near-real-time ingestion . Experience with CI/CD
and infrastructure/configuration automation using Git-based tools. Experience with
Tableau, BI datasets, or data warehouse integration. Databricks, AWS, Data Engineering, AI/ML, or GenAI certifications .
We are looking for a
Data Enginee r to design, build, and support reliable data ingestion and transformation pipelines on
AWS
and
Databricks . The role will work with architects, engineers, QA, and business teams to deliver data pipelines, curated datasets, data-quality controls, and production-ready operational processes. Key responsibilities
Design and build batch and/or streaming data pipelines
using
Databricks
and
AWS
services. Develop data processing across
Bronze, Silver, and Gold layers
using
SQL, Python/PySpark, and Delta Lake . Implement ingestion from APIs, files, databases, object storage, or other enterprise data sources. Develop data transformations, dimensional or curated datasets, and reusable data-engineering components. Implement data validation, reconciliation, error handling, logging, rerun/recovery, and monitoring. Optimise Spark jobs, clusters, partitioning, and storage
for performance and cost. Support CI/CD, environment promotion, SIT/UAT, production deployment, documentation, and operational handover. Collaborate with platform, application, BI, and data-governance teams. About you
3+ years of experience
in data engineering or enterprise data-platform delivery. Hands-on experience with Databricks, Apache Spark, PySpark, and SQL. Experience with Delta Lake
and layered data architecture such as
Bronze/Silver/Gold . Experience building production data pipelines
with proper logging, error handling, and recovery. Experience with AWS services
such as
S3, IAM, Glue, Lambda, Kinesis , or related services. Good understanding of data modelling, data quality, and performance optimisation. Experience with Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, or Structured Streaming. Experience with APIs, XML/JSON, event or near-real-time ingestion . Experience with CI/CD
and infrastructure/configuration automation using Git-based tools. Experience with
Tableau, BI datasets, or data warehouse integration. Databricks, AWS, Data Engineering, AI/ML, or GenAI certifications .