Technical Lead Data Engineering_Sparity_Hyderabad

Dear All,

Greetings From Sparity..!!

Role: Technical Lead Data Engineering

Location: Offshore – India

Experience: 9 to 12 yrs

Tech Stack: AWS (Glue, Lambda, Step Functions, RDS, Aurora PostgreSQL, DynamoDB), PySpark, Python, SQL, Oracle, MongoDB, Streaming and Batch Data Integration, AI-Assisted Development

Working Knowledge: EKS, ECS, Docker, OpenShift, VPC, RDS, Security Groups, Terraform, GitHub, Jenkins, GitHub Actions, encryption, IAM

Role Overview

We are looking for a Lead Data Engineer to own the technical direction and delivery of data integration and data engineering work on AWS. Working alongside the Backend and Integration, UI/UX, and Automation teams, you will set engineering standards for batch and streaming pipelines, design for error handling and replay, and scale relational and NoSQL stores to multi-terabyte volumes. This role combines deep hands-on ownership with team leadership and acts as the primary technical escalation point for data pipeline design, performance, and reliability.

Key Responsibilities

• Lead the design, build, and delivery of ETL/ELT pipelines using AWS Glue and PySpark for batch and streaming data integration
• Design error handling, retry, and replay patterns so that pipelines recover cleanly and data is never silently lost
• Own data modeling and storage decisions across Aurora PostgreSQL, Oracle, MongoDB, DynamoDB, and RDS
• Lead the scaling of RDBMS and NoSQL systems to multi-terabyte volumes, including partitioning, indexing, and performance tuning
• Write and optimize complex SQL queries and stored procedures supporting reporting and downstream applications
• Perform data analysis and profiling to validate data quality, completeness, and consistency across pipelines
• Orchestrate pipelines using Lambda and Step Functions, and coordinate with the Backend and Integration team on events and APIs
• Define coding standards, design patterns, and best practices for data engineering across the team
• Own SDLC quality across design, configuration, testing, debugging, implementation, and support, and work with the Automation Engineer on data validation tests
• Apply working knowledge of ECS, Terraform, GitHub, Jenkins, GitHub Actions, IAM, and encryption to deploy and secure pipelines
• Own production incident response and root-cause analysis for data pipeline issues, and implement long-term reliability improvements
• Mentor Senior Software Engineers, conduct code reviews, and document data flows and pipeline architecture

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field
• 9 to 12 years of experience in data engineering or ETL development, with demonstrated progression toward technical leadership
• Expert, hands-on experience with AWS Glue and PySpark (must haves)
• Strong knowledge of data integration patterns: streaming, batch, and error and replay
• Strong hands-on experience with Oracle, MongoDB, and DynamoDB (must haves), and with Aurora PostgreSQL and RDS
• Proven experience building and scaling RDBMS (PostgreSQL, Aurora PostgreSQL, Oracle) and NoSQL (MongoDB, DynamoDB) systems to multi-terabyte scale
• Hands-on knowledge of SQL and stored procedures, and strong data analysis techniques
• Strong proficiency in Python for scripting, automation, and Glue/PySpark development
• Experience across the full software development lifecycle: design, configuration, testing, debugging, implementation, and support of large-scale, business-centric, process-based applications
• Working awareness of EKS, ECS, Docker, OpenShift, VPC, RDS, Security Groups, Terraform, GitHub, Jenkins, GitHub Actions, encryption, and Identity and Access Management (IAM); working knowledge of ECS and Terraform is a must have
• Strong debugging, performance-tuning, and production-support skills across complex pipelines
• Experience leading technical projects or guiding a team of data engineers
• Excellent written and verbal communication skills for cross-functional and stakeholder engagement

AI Knowledge & AI-Assisted Development (Required)

• Good understanding of AI/LLM models and how they can be applied to accelerate delivery, including prompt engineering
• Daily, practical use of AI development tools such as Devin AI, Cursor, and Claude Skills to accelerate development of Glue/PySpark scripts and SQL
• Able to critically review and validate AI-generated code, queries, and transformations for correctness, performance, and data integrity before deployment
• Defines and enforces team-level standards for responsible AI-assisted development, including validation rigor, security review of AI-generated code, and prompt hygiene
• Practical use of AI tools to assist with data profiling, anomaly detection, and technical documentation
• Understanding of secure and compliant AI tool usage, including never entering PHI, member data, client-specific credentials, or other sensitive information into prompts or external AI tools

Preferred Qualifications

• Experience in the US health insurance or payer domain: claims, eligibility, enrollment, provider, or member data, with HIPAA-aware system design and data handling practices
• Exposure to healthcare interoperability and data standards (HL7, FHIR, X12 EDI)
• Experience with additional AWS data services such as Athena, Redshift, Kinesis, or EMR
• Experience with modern orchestration tools (for example Apache Airflow)
• Relevant certifications: AWS Certified Data Engineer or Data Analytics Specialty

If interested, please apply here OR or share your CV to saikumar.sangu@sparity.com