SDE IV
Posted Updated
Description
Responsibilities:
- Design and Development: Bridge OSS tools (Spark, Flink, Airflow, Iceberg), internal services, and cloud offerings into a cohesive data platform infrastructure. Build intuitive platform integrations enabling push-button data workflows.
- Scale Engineering: Operate distributed systems processing petabytes of data daily. Own multi-region Kubernetes infrastructure with elastic scalability and fault tolerance.
- Performance Optimisation: Optimise compute utilisation (Spark/Flink clusters, Velox/Gluten acceleration) for large-scale batch and real-time streaming with sub-second latency.
- Observability and Data Quality: Build comprehensive telemetry (metrics, logs, traces) and data quality frameworks for 24/7 uptime. Enforce SLAs/SLOs with automated incident response and data validation.
Requirements:
- 7-10 years building, optimising, and operating production data platforms.
- Deep data engineering fundamentals: data modelling, partitioning strategies, query optimisation.
- Distributed computing: Spark (PySpark/Scala), Flink streaming, performance tuning at petabyte scale.
- Data lake architecture: Iceberg table format, Polaris catalogue, schema evolution, time travel.
- Orchestration: Airflow DAG development, dependency management, and SLA monitoring.
- Data transformation: DBT modelling, testing, documentation, incremental builds.
- Data quality: Great Expectations, dqueue validation frameworks, and drift detection.
- Query acceleration: Velox, Gluten integration, columnar formats (Parquet, ORC).
- Data governance: OpenMetadata catalogue, lineage tracking, and access control.
- Kubernetes platform development: operators (Spark/Flink), Yunikorn scheduler, multi-tenancy, and autoscaling.
- Cloud infrastructure: GKE multi-region clusters, GCS object storage, hybrid cloud/on-prem architecture.
- Programming: Python, PySpark, Scala for data pipelines and platform tooling.
- IaC: Terraform, Helm, GitOps for reproducible deployments.
- CI/CD: Automated testing, deployment pipelines for data platform components.
Good-to-Have:
- Experience in building a cloud data platform/control plane development.
- Advanced observability: Prometheus/Grafana, Loki, Firehydrant integration.
- Real-time streaming: Kafka integration, exactly-once semantics, backpressure handling.
- Cost optimisation: Resource allocation, query optimisation, storage tiering strategies.
Skills
- Airflow
- CI/CD
- Cloud
- Data Engineering
- Data Governance
- Data Lake
- Data Modeling
- Data Pipelines
- Data Quality
- dbt
- Distributed Computing
- Distributed Systems
- Flink
- GitOps
- GKE
- Grafana
- Helm
- Infrastructure as Code
- Kafka
- Kubernetes
- Loki
- Observability
- Parquet
- Prometheus
- PySpark
- Python
- Scala
- Spark
- Terraform
- Test Automation
Sign in to read and join the discussion.