Lead Data Engineer
Posted
Description
SNI is serving as a trusted IT Outsourcing partner in line with the needs of World's most prestigious firms and carried out successful projects worldwide.
Responsibilities:
- Lead data engineering work streams and hands-on technical implementation across multi-functional squads.
- Design and deploy resilient, scalable batch and streaming pipelines and data processing frameworks.
- Drive platform migrations from legacy schedulers to modern, cloud-native workflow orchestration tools.
- Establish data governance policies, lineage tracking, metadata management, and role-based security controls (e.g., Unity Catalog).
- Implement data virtualization and unified integration patterns across disparate, multi-source enterprise systems.
- Lead performance tuning, query optimization, cost management, and operational reliability across the data platform.
Skills:
- 6–9 years of professional experience in data engineering and enterprise data platform delivery.
- Expert-level hands-on experience with Databricks and Apache Spark.
- Deep knowledge of Data Lakehouse architecture (Delta Lake, medallion architecture, schema enforcement).
- Advanced proficiency in Python, SQL, and PySpark for data transformation and distributed computing.
- Strong background in workflow orchestration tools (e.g., Airflow, Databricks Workflows, Dagster) and scheduler migration projects.
- Practical experience implementing data governance, cataloging, metadata, and data access security frameworks.
- Proven track record architecting data virtualization and integration solutions across heterogeneous cloud and on-premise sources.
Skills
Sign in to read and join the discussion.