Data Engineer, Databricks & DevOps for Scalable Pipelines
Posted Updated
COGNIZANT TECHNOLOGY SOLUTIONS ASIA PACIFIC PTE. LTD. Data Engineer, Databricks & DevOps for Scalable Pipelines
Description
Cognizant Technology Solutions Asia Pacific PTE. LTD. is seeking a Data Engineer to design, build, and maintain scalable data pipelines on AWS and Databricks. The role combines data engineering with DevOps practices to deliver automated, secure data platforms supporting analytics and ML.
Responsibilities include design of batch/streaming pipelines, lakehouse architecture, and orchestration with Airflow or Databricks Workflows. Strong Python/SQL, AWS, and CI/CD experience are essential.
Design and build scalable data pipelines on AWS and Databricks. Develop batch and streaming ETL/ELT workflows. Implement lakehouse architectures with Delta Lake and Unity Catalog. Orchestrate workflows using Airflow or Databricks Workflows. Provision cloud infrastructure with Terraform or CloudFormation. Build CI/CD pipelines for data code and infra. Monitor pipelines and platform health (CloudWatch, Datadog). Optimize Spark jobs and cluster configs. Enforce data quality, governance, security, and access controls. Collaborate with data scientists, analysts, and stakeholders. Document pipelines, architecture, and runbooks. Bachelor's degree in CS, Engineering, or related field. Hands-on Databricks experience (PySpark, Delta Lake, Jobs/Workflows, cluster management). Strong AWS skills: S3, IAM, Glue, Lambda, EC2, Kinesis/MSK, Redshift, VPC, CloudWatch. Proficiency in Python and SQL. Experience with Git, CI/CD pipelines, Docker, and infrastructure as code (Terraform preferred). Understanding of data modeling, warehousing, and lakehouse concepts. Experience with Airflow and Databricks Workflows. Familiarity with Agile/Scrum delivery. Databricks Workflows CloudWatch MonitoringSkills
- Agile
- Airflow
- Analytics
- AWS
- CI/CD
- Cloud
- CloudFormation
- CloudWatch
- Data Engineering
- Data Modeling
- Data Pipelines
- Data Quality
- Databricks
- Datadog
- Delta Lake
- DevOps
- Docker
- EC2
- ELT
- ETL
- Git
- IAM
- Infrastructure as Code
- Kinesis
- Lakehouse
- Lambda
- Machine Learning
- PySpark
- Python
- Redshift
- S3
- Scrum
- Spark
- SQL
- Terraform
- Unity
- Unity Catalog
- VPC
Sign in to read and join the discussion.