Sr. Pyspark Data Engineer
Job Title: PySpark Data Engineer
Experience: 4+ Years
Location: Hyderabad
Job Summary:
We are seeking a Senior Spark Engineer to design and implement high-performance Spark execution patterns inside xFlows, supporting batch and streaming pipelines with built-in data quality, observability, and governance.
Requirements
Key Responsibilities
- Design and implement Spark-based execution frameworks for xFlows pipelines.
- Build reusable Spark components for:
- Readers (JDBC, Files, Kafka, CDC)
- Transformers (Join, Filter, Aggregate, Window, Union)
- Writers (Iceberg, Delta, Parquet, Snowflake)
- Readers (JDBC, Files, Kafka, CDC)
- Optimize Spark performance (partitioning, caching, shuffles, memory).
- Implement Data Quality & Reconciliation execution patterns.
- Handle schema evolution, CDC, watermarking, and checkpoints.
- Integrate Spark jobs with EMR Serverless / Data bricks / Kubernetes.
- Publish execution metrics, logs, and lineage for observability.
- Work closely with platform & UI teams to support no-code execution.
Required Skills
- 4+ years of experience with Apache Spark
- Strong expertise in PySpark (preferred) or Spark Scala
- Deep understanding of Spark internals (DAGs, stages, shuffles, caching)
- Experience with Iceberg / Delta Lake
- Strong SQL skills
- Experience with batch and streaming pipeline
Benefits
- Comprehensive Medical Coverage:Health insurance of INR 7.0 Lakhs for you and your family (up to 6 members), ensuring complete peace of mind.
- Robust Protection Plans:Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones.
- Retirement Benefits:PF and Gratuity provided as per standard government regulations.
- Flexible Work Options:Enjoy hybrid work arrangements & flexible working hours
- Generous Leave Policy:21 days of annual leave, in addition to 10 company-declared holidays.
- Employee Well-being Spaces:Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.