Data Engineer
Posted Updated
Description
Data Engineer - Cloud & Big Data
Role Summary:
Responsible for designing, developing, and maintaining scalable cloud-based data pipelines and data platforms across AWS, GCP, Kafka, and Databricks. The role involves building reliable ETL/ELT pipelines, real-time data ingestion, data transformation, monitoring, and performance/cost optimization.
Responsible for designing, developing, and maintaining scalable cloud-based data pipelines and data platforms across AWS, GCP, Kafka, and Databricks. The role involves building reliable ETL/ELT pipelines, real-time data ingestion, data transformation, monitoring, and performance/cost optimization.
In this role you will...
- Design and develop ETL/ELT data pipelines using AWS Glue, Databricks, and Spark.
- Build and maintain batch and real-time data pipelines using Apache Kafka / Confluent Kafka.
- Develop CDC-based data ingestion and event-driven data processing.
- Work with AWS services including Glue, S3, Lambda, IAM, KMS, Secrets Manager, and related services.
- Develop and optimize data solutions on GCP, including BigQuery, Cloud Storage, Cloud Functions, and related services.
- Implement data processing using Databricks, PySpark, Delta Lake, Bronze/Silver/Gold architecture, and DLT.
- Manage and optimize Elasticsearch/OpenSearch data ingestion, indexing, queries, and cluster performance.
- Implement data quality, validation, monitoring, logging, and alerting across data pipelines.
- Troubleshoot production issues involving Kafka, schema compatibility, CDC, Spark, Glue, Databricks, and cloud infrastructure.
- Optimize pipeline performance and cloud infrastructure costs.
- Implement secure data solutions using IAM, encryption, network controls, secrets management, and least-privilege access.
- Collaborate with application, DevOps, QA, analytics, and product teams to deliver reliable data solutions.
You'vegot what it takes if you have...
- 5+ years of experience in Data Engineering / Big Data Engineering.
- Strong hands-on experience with AWS + Databricks + Kafka.
- Experience working in multi-cloud environments (AWS/GCP).
- Strong understanding of distributed systems, streaming, CDC, schema evolution, and data governance.
- Experience troubleshooting high-volume production data pipelines.
- Experience with cloud performance and cost optimization.
#LI-Onsite