Data Engineering Manager - Data Platforms

Job Overview:

We are looking for a Data Engineering Manager with 10 - 15 years of experience to lead data engineering initiatives, build scalable data pipelines, manage data lake architecture, and enable real-time analytics and AI use cases. The role will work closely with application, ML, analytics, and business teams while mentoring data engineers and driving modern engineering practices.

Key Responsibilities:

- Lead real-time pipelines using Kafka/MSK, Debezium CDC, and Apache Pinot.

- Design and maintain Java/Python-based frameworks for ingestion, validation, transformation, and replication.

- Design and manage data lakes using Apache Iceberg, AWS Glue, Athena, and S3.

- Define data modeling standards across Pinot, Redshift, and warehouse layers.

- Lead orchestration and workflow platforms using Apache Airflow and Superset.

- Drive cloud-native deployments using ECS/EKS, Docker, Terraform/CDK, and CI/CD.

- Support ML, analytics, product, and AI initiatives with reliable data solutions.

- Ensure data quality, observability, lineage, governance, and security across the data stack.

- Mentor data engineers and promote modern architecture and engineering best practices.

Required Skills:

- Real-Time Data: Kafka/MSK, Debezium, Event-Driven Architecture, CDC

- Data Platforms: Apache Pinot, Redshift, RocksDB/NoSQL, Apache Iceberg, Athena, AWS Glue

- Programming: Java, Python

- Orchestration: Apache Airflow, Superset

- AWS: S3, IAM, Lambda, Glue, ECS/EKS, CloudWatch

- Data Architecture: Data Lakes, Streaming & Batch Processing, Data Modeling, Parquet, Avro

- DevOps: Docker, Terraform/CDK, CI/CD, GitHub Actions

Good to Have:

- dbt, Trino/Presto, ClickHouse, or Druid.

- Data security, encryption at rest/in transit, GDPR/PCI compliance.

- CloudFormation and advanced DevOps practices.

- Experience with scalable cloud-native data architectures.