DataOps/Cloud Data Engineer - Senior

Position Overview

The Senior DataOps / Cloud Data Engineer will be responsible for designing, developing, implementing, and optimizing enterprise-scale data pipelines and cloud data platforms. The role requires strong hands-on experience with Databricks, Delta Live Tables (DLT), Unity Catalog, Azure Data Factory, PySpark, advanced SQL, and enterprise data migration.

The successful candidate will build and maintain scalable ETL/ELT processes, develop unified data platforms, implement data integration solutions, and migrate large volumes of data from traditional relational database environments to modern cloud platforms, including Azure Data Lake, Databricks, and Microsoft Fabric.

The candidate will also contribute to DataOps automation, CI/CD implementation, data governance, performance optimization, and reliable data provisioning across enterprise data environments.

Key Responsibilities

1. Data Engineering and ETL/ELT Development

• Design, develop, automate, and optimize complex ETL/ELT processes using Databricks, Delta Live Tables (DLT), and equivalent technologies.
  
• Build scalable and reliable data pipelines to ingest, transform, process, and provision data from multiple source systems.
  
• Develop custom data pipelines using Databricks and implement efficient data processing workflows.
  
• Optimize data pipeline performance, reliability, scalability, and operational efficiency.
  
• Support structured, semi-structured, and unstructured data ingestion and processing.
  

2. Cloud Data Platforms and Lakehouse Architecture

• Design and develop unified enterprise data platforms using Databricks and Unity Catalog or equivalent technologies.
  
• Develop and maintain cloud data solutions, including data lakes, lakehouses, and enterprise data warehouses.
  
• Implement medallion architecture patterns, including Bronze, Silver, and Gold data layers.
  
• Design and implement relational and dimensional data models, including fact and dimension tables.
  
• Support data governance, cataloguing, access management, and secure data provisioning.
  

3. Azure Data Engineering

• Develop and manage Azure Data Factory pipelines, including copy activities and custom data integration workflows.
  
• Integrate Azure Data Factory with Databricks and other enterprise data platforms.
  
• Configure and maintain cloud data services, storage, repositories, and associated infrastructure.
  
• Support data migration and integration across Oracle, Azure Data Lake, Databricks, Azure SQL, and Microsoft Fabric.
  
• Apply Azure DevOps Git integration and CI/CD practices to data pipeline development and deployment.
  

4. Database Development and Optimization

• Develop and optimize advanced SQL queries using PL/SQL, T-SQL, and equivalent technologies.
  
• Work with relational databases, including Oracle, MySQL, and Microsoft SQL Server.
  
• Perform data modelling, data mapping, data replication, extraction, transformation, and loading.
  
• Troubleshoot database and pipeline performance issues and implement optimization strategies.
  
• Support database administration activities, including operational troubleshooting, failover, recovery, and maintenance as required.
  

5. Big Data Processing and Data Migration

• Develop large-scale data processing solutions using PySpark and other applicable big data frameworks.
  
• Design and implement enterprise data migration strategies for high-volume data environments.
  
• Migrate and integrate data from Oracle databases to Azure Data Lake, Databricks, and Microsoft Fabric.
  
• Support migration of OLTP and OLAP workloads to cloud-based SaaS, PaaS, and IaaS environments.
  
• Ensure data accuracy, consistency, integrity, and reliability throughout migration and integration processes.
  

6. DataOps, CI/CD, and Automation

• Implement DataOps principles, standards, and best practices across data engineering workflows.
  
• Automate data pipeline deployment, orchestration, testing, and operational processes.
  
• Implement continuous integration and continuous delivery/deployment (CI/CD) using Azure DevOps and Git integration.
  
• Monitor pipeline performance, identify bottlenecks, and implement continuous improvements.
  
• Support production operations, incident resolution, and reliable delivery of data services.
  

7. Collaboration and Delivery

• Collaborate with technical teams, business stakeholders, and other project participants to deliver enterprise data solutions.
  
• Apply Agile development methodologies and established engineering practices.
  
• Demonstrate strong analytical, problem-solving, and decision-making capabilities.
  
• Communicate technical concepts effectively through documentation, presentations, and stakeholder discussions.
  
• Maintain delivery commitments and meet project deadlines.
  

Requirements

Mandatory Skills and Qualifications

Candidates must have at least 8 years of relevant experience and demonstrate hands-on expertise in the following areas:

1. Databricks and ETL/ELT: Design, build, automate, and optimize complex data pipelines using Databricks, Delta Live Tables (DLT), or equivalent technologies.
   
2. Unity Catalog and Unified Data Platforms: Develop unified data platforms using Databricks and Unity Catalog or equivalent solutions.
   
3. Relational Databases: Strong experience with Oracle, MySQL, and SQL Server, including database development, administration, data replication, and query optimization.
   
4. Advanced SQL: Proficiency in PL/SQL and T-SQL for complex data extraction, transformation, and processing.
   
5. Big Data Processing: Hands-on experience with PySpark and distributed data processing.
   
6. Azure Data Factory: Experience creating and managing Azure Data Factory pipelines, including copy activities and custom data integration workflows.
   
7. CI/CD and Azure DevOps: Experience implementing CI/CD pipelines for Databricks and data engineering solutions, including Azure DevOps Git integration.
   
8. Data Migration and Integration: Demonstrated experience migrating and integrating data across platforms, particularly:
   
• Oracle to Azure Data Lake
  
• Oracle to Databricks
  
• Oracle to Microsoft Fabric
  

Desirable Skills and Experience

• Experience designing and implementing data lakes and lakehouses using medallion architecture.
  
• Advanced hands-on experience with Oracle databases, Azure SQL Server, and Azure Data Factory.
  
• Knowledge of cloud platforms, including Microsoft Azure, Google Cloud Platform (GCP), and Amazon Web Services (AWS).
  
• Experience with PL/SQL-based ETL development, data extraction, transformation, and operational support.
  
• Experience in troubleshooting, performance tuning, failover, and recovery.
  
• Proficiency in scripting and programming languages, including Python, Unix Shell scripting, and Scala.
  
• Experience with enterprise data warehouses, data governance, data security, and data provisioning.
  
• Familiarity with Agile delivery, DataOps practices, and automated deployment processes.
  

Education

Educational qualifications are evaluated as part of the overall candidate assessment. Specific degree or certification requirements were not explicitly stated in the requisition.

Public Sector Experience

Experience working with government or public sector organizations is desirable and will be considered as part of the candidate evaluation.