Data Engineer (Databricks & DevOps)

We are looking for a Data Engineer to design, build, and maintain scalable data pipelines on AWS and Databricks. You will combine data engineering with DevOps practices (CI/CD, infrastructure as code, monitoring) to deliver reliable, automated, and secure data platforms that support analytics, reporting, and machine learning.

Key Responsibilities

• Design and develop batch and streaming data pipelines using Databricks (PySpark, Spark SQL, Delta Lake) on AWS
• Build and optimize ETL/ELT workflows to ingest data from databases, APIs, files, and event streams
• Implement lakehouse architecture using Delta Lake, Unity Catalog, and the medallion model (bronze/silver/gold)
• Manage and orchestrate workflows with Databricks Workflows, Airflow, or AWS Step Functions
• Provision and manage cloud infrastructure using Terraform or CloudFormation
• Build and maintain CI/CD pipelines for data code and infrastructure (GitHub Actions, GitLab CI, Jenkins, or Azure DevOps)
• Implement monitoring, logging, and alerting for pipelines and platform health (CloudWatch, Datadog, etc.)
• Optimize Spark jobs and cluster configurations for performance and cost
• Enforce data quality, governance, security, and access control standards
• Collaborate with data scientists, analysts, and business stakeholders to translate requirements into technical solutions
• Document pipelines, architecture, and operational runbooks

Required Skills and Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field
• Strong hands-on experience with Databricks (PySpark, Delta Lake, Jobs/Workflows, cluster management)
• Solid experience with AWS services: S3, IAM, Glue, Lambda, EC2, Kinesis/MSK, Redshift, VPC, CloudWatch
• Proficiency in Python and SQL
• Experience with DevOps tools and practices: Git, CI/CD pipelines, Docker, and infrastructure as code (Terraform preferred)
• Understanding of data modeling, data warehousing, and lakehouse concepts
• Experience with workflow orchestration tools (Airflow, Databricks Workflows)
• Familiarity with Agile/Scrum delivery

Preferred Qualifications

• Databricks Certified Data Engineer (Associate/Professional)
• AWS certifications (Solutions Architect, Data Engineer, or DevOps Engineer)
• Experience with Unity Catalog, DLT (Delta Live Tables), and Auto Loader
• Knowledge of Kubernetes, Kafka, or dbt
• Experience with data governance, lineage, and compliance (GDPR, PDPA, etc.)
• Exposure to MLOps or supporting ML workloads
• Scripting skills in Bash or PowerShell