Senior Data Engineer

Specific Accountability :-

• Lead the design, build, and optimization of batch and real-time data pipelines.
• Own data model design, ingestion frameworks, transformation logic, and platform reliability.
• Ensure high standards for data quality, lineage, governance, security, and compliance.
• Drive performance tuning, cost optimization, monitoring, and incident resolution across data platforms.

Translate business and analytics needs into scalable technical solutions and delivery plans.

Guide code reviews, technical standards, and release practices for data engineering work.

Mentor team members, allocate work, and provide technical leadership on complex initiatives.

Collaborate with value delivery owners, architects, analysts, scientists, and application teams to enable trusted data products

Desired Candidate Profile

Experience level: 10+ years hands-on data engineering with production pipeline ownership; at least one platform or system owned end to end.

Data transformation and modeling

• dbt (data build tool): models, tests, macros, incremental models, project structure
• Data modeling: dimensional modeling, star schema, medallion architecture (raw / curated / presentation, or bronze / silver / gold)

Data warehousing and lakehouse architecture concepts

• Data warehousing and lakehouse architecture concepts

Orchestration

• Apache Airflow: DAG design, task dependencies, triggers, sensors, parameterized runs, failure handling, scheduling

Workflow orchestration and monitoring of production pipelines

• Workflow orchestration and monitoring of production pipelines

Data ingestion and loading

• Batch data ingestion and ETL / ELT pipeline development
• Load patterns: full load, incremental load, upsert / merge, change data capture (CDC)
• Multi-source ingestion: SFTP, REST APIs, relational databases, file-based ingestion
• Data quality, data validation, schema evolution handling

Query engine, storage, and table formats

• Dremio (or equivalent lakehouse query engine: Trino, Presto, Starburst, Athena)
• Apache Iceberg (or equivalent open table format: Delta Lake, Apache Hudi)
• S3 / S3-compatible object storage (AWS S3, MinIO, Dell ECS)
• Partitioning, table layout, and query performance tuning

Programming and databases

• Python (pipeline development, automation, scripting)
• Advanced SQL, query optimization, performance tuning
• PostgreSQL and/or SQL Server (relational databases)

Non-relational / NoSQL databases (must-have, not just exposure)

The platform ingests from and operates alongside several non-relational stores. The candidate must be able to work with them as a data engineer, meaning read from them, model their data into the lakehouse, and reason about their consistency and performance characteristics. Screen for hands-on experience with at least two of the following families:

• Key-value / in-memory stores: Redis (caching, queues, state; TTL and eviction behaviour; using Redis as a pipeline coordination or lookup layer)
• Document databases: MongoDB, Couchbase, or DocumentDB (nested JSON documents, schema-on-read, flattening documents into relational / lakehouse tables)
• Wide-column stores: Cassandra, ScyllaDB, or HBase (partition-key design, read patterns, extracting at scale)
• Search / analytics engines: Elasticsearch or OpenSearch (index design, querying, extracting data out of an index for analytics)

Data formats and semi-structured data

• Columnar formats: Apache Parquet (required), ORC; row formats: Avro
• Text and semi-structured formats: JSON, JSON Lines, CSV / delimited files, XML; encoding, delimiter, and quoting edge cases
• Compression codecs (Snappy, gzip, zstd) and their effect on storage and query
• Handling nested and evolving schemas end to end (schema inference, drift detection, schema evolution: column add / delete / type change)

Distributed processing

• Apache Spark / PySpark (DataFrame API, partitioning, joins, skew handling, writing Iceberg / Parquet); or equivalent distributed engine
• Understanding of parallelism, shuffles, memory tuning, and cost of distributed jobs

Scripting, Linux, and connectivity

• Bash / shell scripting and Linux command line (the deployment and operational tooling is script-based)
• SFTP / SSH: key management, automated transfers, file-based handoffs (verify / result files, archiving, retries)
• TLS certificates and secure connections (understanding certificate verification; never disabling it in production)
• Connectivity troubleshooting: DNS, firewalls, proxies, connection timeouts, service-to-service networking inside Kubernetes / OpenShift
• YAML and configuration management for pipelines, dbt, Airflow, and Kubernetes manifests