Data Engineer

Data Engineer – Azure / AWS / Databricks / Microsoft Fabric

About the Role

We are looking for a Data Engineer with 2–3 years of experience to join our Data & Analytics team and contribute to the development of modern, cloud-based data platforms and analytics solutions.

The role involves designing and developing scalable data pipelines, data lake/lakehouse solutions, data transformations and curated data layers that support Business Intelligence, Advanced Analytics, AI/ML and Decision Intelligence use cases.

The engineer will work closely with Data Architects, Business Analysts, BI Developers, Data Scientists and Project Managers to convert business requirements into reliable and production-ready data solutions.

Key Responsibilities

• Design, develop and maintain scalable ETL/ELT data pipelines.
  

• Build data ingestion and transformation pipelines across databases, APIs, enterprise applications and file-based sources.
  

• Develop data processing solutions using SQL, Python and PySpark/Spark.
  

• Work with cloud data platforms, Data Lakes and Lakehouse architectures.
  

• Develop and maintain Bronze, Silver and Gold/curated data layers.
  

• Implement data transformations, cleansing, validation, reconciliation and data quality checks.
  

• Work with Databricks, Microsoft Fabric and/or Azure data services.
  

• Develop efficient and reusable data engineering frameworks and components.
  

• Optimize SQL queries, Spark jobs and data pipelines for performance and cost.
  

• Implement appropriate error handling, logging, monitoring and pipeline recovery mechanisms.
  

• Support data modelling for analytical and reporting use cases.
  

• Collaborate with BI teams to provide trusted and analytics-ready datasets.
  

• Support Data Scientists and AI teams with appropriately prepared datasets and feature-ready data.
  

• Follow engineering practices around Git, version control, CI/CD, testing and deployment.
  

• Contribute to technical documentation, data lineage and operational runbooks.
  

• Troubleshoot production data pipeline and data quality issues.
  

• Participate in code reviews and follow development standards.
  

• Work in an Agile project environment and collaborate with distributed teams and client stakeholders.
  

Typical Technology Environment

The team works across modern cloud and data technologies including:

• Microsoft Azure
  

• Microsoft Fabric
  

• Databricks
  

• Apache Spark / PySpark
  

• SQL
  

• Python
  

• Delta Lake
  

• Data Lakes / Lakehouses
  

• Azure Data Factory / Fabric Data Factory
  

• Power BI
  

• Git / Azure DevOps
  

• Relational and NoSQL databases
  

• APIs and enterprise data sources
  

The exact technology stack may vary by project.

Requirements

Requirements

Required

• Bachelor's degree in Computer Science, Information Technology, Engineering, Data Engineering or a related technical discipline.
  

• 2–3 years of professional experience in Data Engineering.
  

• Strong hands-on experience with SQL.
  

• Good programming experience in Python.
  

• Experience developing and supporting ETL/ELT pipelines.
  

• Understanding of Data Lake, Lakehouse or modern data platform architectures.
  

• Experience working with structured and semi-structured data.
  

• Good understanding of data transformation, cleansing, validation and data quality concepts.
  

• Experience working with relational databases.
  

• Understanding of dimensional modelling and analytical data structures.
  

• Good understanding of data engineering best practices including logging, error handling, testing and monitoring.
  

• Experience working in an Agile development environment.
  

• Ability to understand business requirements and translate them into technical data solutions.
  

Preferred

• Hands-on experience with Microsoft Azure.
  

• Experience with Databricks.
  

• Experience with Apache Spark / PySpark.
  

• Experience with Microsoft Fabric.
  

• Experience with Azure Data Factory or Fabric Data Factory.
  

• Experience with Delta Lake / Delta Tables.
  

• Exposure to Apache Iceberg or other open table formats.
  

• Experience with Git and CI/CD.
  

• Experience with Power BI or other BI platforms.
  

• Experience integrating data from enterprise applications such as SAP, Salesforce, ERP, CRM, LIMS or similar systems.
  

• Exposure to REST APIs and API-based data ingestion.
  

• Exposure to streaming/event-driven data pipelines.
  

• Understanding of data governance, security, access control and PII handling.
  

• Exposure to AI/ML data pipelines or analytics platforms.
  

Education

Bachelor's degree in Computer Science, Information Technology, Engineering or a related technical field.

Experience

2–3 years of relevant professional Data Engineering experience.

Candidates with strong hands-on project experience in SQL, Python, Spark and cloud data engineering may also be considered where their experience is slightly outside the stated range.