Data Engineer Lead - W2
Posted Updated
Description
Data Engineer Lead - W2
As a Data Engineer Lead, you will play a pivotal role in designing, developing, and managing large-scale data pipelines and data lake patterns. You will lead a team of data engineers, driving the adoption of enterprise-authorized AI-assisted engineering practices to improve code quality, delivery speed, and operational outcomes. You will be responsible for building robust ETL/ELT pipelines, optimizing distributed processing workloads, and ensuring data quality, governance, and compliance.
Responsibilities:
- Execute creative software solutions, design, development, and technical troubleshooting with the ability to think beyond routine or conventional approaches to build solutions or breakdown technical problems.
- Drive team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes.
- Lead architecture and hands-on delivery of large-scale data pipelines (ETL/ELT) covering ingestion, transformation, validation, reconciliation, and publishing across curated layers.
- Design and operationalize data lake patterns, including partitioning strategies, data quality controls, lineage, governance, and reusable datasets/data products.
- Build and optimize distributed processing workloads using Apache Spark and modern storage/file formats (e.g., Parquet, Avro).
- Drive performance tuning across compute and storage (e.g., Spark tuning: shuffle, joins, caching, skew handling; and warehouse tuning where applicable).
- Implement engineering best practices: code quality, automated testing, CI/CD, observability (metrics/logs/traces), security-by-design, and operational readiness.
- Build and maintain Java-based services and components that support data workflows (e.g., ingestion services, orchestration helpers, APIs, data access layers).
- Develop REST APIs and integration components to enable downstream consumption and platform interoperability.
- Apply modern application engineering practices (clean architecture, SOLID principles, test automation, and secure coding practices).
- Apply knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.
Requirements:
<>Required Skills:- Formal training or certification on software engineering concepts and 5+ years applied experience.
- Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.
- Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices.
- 10+ years of software engineering experience, with strong depth in data engineering/big data platforms.
- Strong hands-on development experience in Java plus proficiency in at least one data-focused language such as Python.
- Proven experience designing and building robust ETL/ELT pipelines and data integration frameworks.
- Strong experience with Apache Spark and distributed processing concepts (fault tolerance, partitioning, performance tuning).
- Strong understanding of data storage serialization and formats such as Parquet and Avro.
- Solid knowledge of data lake/lakehouse concepts and patterns (e.g., Medallion architecture), including data quality, metadata, and governance considerations.
- Experience working with cloud data warehouses such as Snowflake (loading/unloading patterns, performance basics) or equivalent.
- Ability to lead technical discussions, communicate clearly to varied stakeholders, and drive delivery in cross-functional environments.
- Experience with orchestration and workflow scheduling tools (e.g., Airflow, Dagster, Control-M, etc.).
- Exposure to streaming/event-driven processing (e.g., Kafka, Spark Structured Streaming) and incremental processing patterns (CDC, upserts).
- Experience with table formats / ACID layers (e.g., Delta Lake, Apache Iceberg, Hudi).
- Strong production operations mindset (on-call practices, SLAs/SLOs, observability, incident response).
- Experience in regulated environments (risk, audit, compliance, privacy, PII handling).
- Experience with cost optimization / FinOps practices for data platforms.
- Experience with working with Agentic AI / Gen AI solutions and capabilities will be an added advantage.
Skills
- Agentic AI
- AI
- Airflow
- API
- Automation
- Avro
- Big Data
- CI/CD
- Clean Architecture
- Cloud
- Control-M
- Dagster
- Data Engineering
- Data Lake
- Data Pipelines
- Data Quality
- Delta Lake
- ELT
- ETL
- Event Driven Architecture
- FinOps
- Generative AI
- Iceberg
- Java
- Kafka
- Lakehouse
- Medallion Architecture
- Observability
- Parquet
- Python
- REST
- Secure Coding
- Snowflake
- Spark
- Test Automation
Sign in to read and join the discussion.