Sr Software Engineer - Data Engineering
Summary
Senior data engineer at Asurion's India Global Capability Center in Pune, building and operating Databricks-based lakehouse pipelines end to end — ingestion (batch, streaming, CDC), bronze/silver/gold Delta Lake layers, data quality, governance and privacy controls — plus datasets supporting analytics, ML and generative AI. Core stack: Databricks, PySpark/Spark SQL, Python, Unity Catalog, dbt, Air
What wearelookingfor:
Experienceandeducation:
- Bachelor's degree in Computer Science, InformationTechnology, Electronics or a related engineering discipline.Amaster's (ME, MTech, MS or MCA) in a computing or quantitative discipline is preferred.
- Threetofiveyearsofprofessionalsoftwareengineering,withademonstrablerecordofshipping andsupportingproductionsystems.
- Production pipelines built and operated at scale, not only prototypes or reports, with a significant component owned end to end including its production behavior.
- Data modellingforanalyticalconsumption,withrealaccountabilityforcorrectness.
- Ingestionbuiltforstreaming orchangedatacapture,andmentoringoflessexperiencedengineers,formallyorinformally
- Experience with data governance, privacy or regulatory reporting, or supplying data to machine learning and generativeAI workloads, is a strong advantage.
Coretechnicalskills:
- Databricks:Delta Lake, Unity Catalog, Lakeflow Declarative Pipelines or an equivalent, Workflows, and cluster and SQLwarehouse configuration.
- PySpark andSparkSQL:partitioning,shuffle and skewbehaviour,caching and queryoptimisationon Databricks.
- AdvancedSQL:windowfunctions, complexjoins,incrementalandmergepatterns, andqueryperformancetuning.
- Python:toaproductionstandard,withattention topackaging,testingandcodestructureratherthannotebooks alone.
- DeltaLake:opentableformatswithDeltaLakeasthedefault,including schemaevolution,timetravelandcompaction.
- Unity Catalog:governance,lineageandaccesscontrol,including classification,maskingandfine-grainedgrants.
- Power BIorDatabricksSQL:dashboardsontopofwhatyoudeliver,withasemantic layerthebusiness readsthenumbers from.
- dbtandAirflow:transformation,testingandorchestrationwithlineageanddependencymanagement,onAirfloworDatabricksWorkflows.
- MySQL and DynamoDB:the operational sources the platform ingests from, including change data capture against MySQLand export and streaming patterns from DynamoDB.
Nice to Have:
- Streaming with Kafka or Kinesis, and change data capture with Debezium against MySQL sources.
- MLflow, feature stores and machine learning operations practice.
- Privacy engineering at scale: tokenisation, pseudonymisation and deletion.
- TerraformandGitHubActions fordata pipelines.
- Snowflake orRedshift,formigration andinteroperabilitywork.
- Databricks Data EngineerAssociate or Professional, or a generative AI credential.
How you work:
- Owns outcomes, not tickets.Turns a half-formed requirement into something running, observable and supportable.
- Raises the bar around them. Reviews carefully and kindly, and mentors the engineers nearby.
- Engineering judgement. Picks the simplest design that meets the requirement, and can say what was left out.
- Clear in writing. Design notes, data dictionaries and handover documentation others can follow.
- Rigorous about correctness. Would delay a dataset rather than publish a wrong number.
Whatyouwillown:
Platformandpipelineengineering:
- Ingestion:batchandstreaming,changedatacapture, schemaevolution,lateandduplicate data,andreconciliationagainstsourcesystems.
- Thelakehouse layers: bronze,silverandgoldDeltaLakelayerswithsensiblepartitioning,clustering,compactionandretention.
- Data products:dimensional and event models, slowly changing dimensions, and semantic definitions that mean the same thing to every consumer.
- Datacontracts:agreedwiththeproducingteamssoupstream changedoesnotbreakdownstream silently.
- Dataqualityasengineering:tests,freshnessandcompletenessmonitoring,anomalydetection,andalertsroutedtoanamedowner.
- Orchestration and recovery:Databricks Workflows orAirflow, with dependency management and backfill designed as a standing capability rather than a rescue operation.
- Platformcost:cluster andwarehousesizing,efficientjobs,andconsumptionmadevisibleforthedomains youown.
Governance,privacyandcompliance:
- Unity Catalog:classification,ownership,lineageandfine-grainedaccesscontrolfortheassetsyouown.
- Privacy controls:personal-datadiscoveryandclassification,maskingandtokenisation,purposelimitation,retention anddeletion.
- Data residency:pipelines and storage designed so data stays in the jurisdiction that requires it, including India's DPDP obligations and the regimes of the markets served.
- Audit evidence:ISO27001andpartnerobligationsforthepipelines youown.
EnablingAIandanalytics consumers:
- Retrieval corpora and embedding pipelines:a first-class deliverable, including chunking strategy, refresh cadence and permission-aware access.
- Evaluation and ground-truth datasets:curated, versioned and reproducible, so retrieval quality is a measured property rather than an impression.
- Machine learningsupport:featureengineering,MLflowexperimentandmodeltracking, andreproducibletraining data.
- Analyticalself-service:documented,trustworthydatasetssoroutinequestions donotqueuebehindthedatateam.
Craft,reviewandproductionownership:
- Design beforebuilding:short,cleardesignnotesfornon-trivialwork,takentoreview,withthetrade-offswrittendown.
- Review asrealwork:giveitpropertimein your week.Itisnotoverhead here.
- Operate what you build:production support for the data platform and its critical pipelines, incident response, and the follow-up actions seen through.
- Observabilityforyourpipelines: dashboards,alertsthatleadtoaction,andlogsacolleague candiagnosefromunaided.
- Mentoring:pairwithlessexperiencedengineers,givereviewfeedbackthatteaches,andhandoverworkthatstretchesthem.
AI-accelerateddevelopmentiscoretohowwework:
- Every engineer here uses an approved assistant such as Claude Code, Cursor, GitHub Copilot or Databricks Assistant as part of normal work, and the expectation runs past autocomplete: a scoped multi-file task, a diff you review properly, and tests and static analysis as the gate.
- Treat a prompt as an egress path. No customer data, no credentials, and nothing outside theAsurion acceptable-use and data-classification standards.
Skills
Sign in to read and join the discussion.