Senior Data Engineer

Role-Senior Data Engineer

Experience - 7 to 15 years

Location- Pune, Hyderabad, Bangalore

Required Skills & Experience

Must Have

• 7-12 years of data engineering experience, with strong recent hands-on responsibility for distributed batch or streaming pipelines.
• Strong hands-on Apache Spark experience using PySpark, Scala, or Spark SQL, including tuning and production troubleshooting.
• Experience building lakehouse pipelines on Google Cloud Storage using Apache Iceberg or a comparable open table format.
• Strong SQL skills with practical experience in profiling, transformations, joins, aggregations, incremental processing, and reconciliation.
• Experience with Apache Kafka and Apache Flink or Spark Structured Streaming.
• Hands-on Apache Airflow experience covering scheduling, dependencies, retries, backfills, and monitoring.
• Experience with database, file, API, event, and CDC ingestion using Kafka Connect, Debezium, Datastream, or comparable tools.
• Understanding of data modeling, schema evolution, data contracts, partitioning, file formats, data quality, lineage, and governance.
• Working knowledge of Git, automated testing, CI/CD, containers, GKE-based execution, Cloud Storage, Workload Identity Federation, and Secret Manager.
• Strong problem solving, documentation, collaboration, and stakeholder communication skills.

Good to Have

• Experience with Polaris, Nessie, REST catalogs, Trino, OpenMetadata, schema registries, Dataproc, or data-quality frameworks.
• Experience with GCP data services and regulated enterprise environments.
• Exposure to banking data, high-volume transaction processing, reconciliation, auditability, or data privacy controls.
• Experience with reusable data-product patterns, data contracts, semantic models, or domain-oriented data platforms.
• Exposure to performance engineering, recovery testing, and multi-tenant GCP data platforms.