Big Data Engineer
Summary
Designs and builds scalable big-data pipelines using Hadoop, Spark, and cloud services (AWS/Azure/GCP) to process and transform large datasets for a data platform.
We are looking for a Senior/Lead Big Data Engineer who can help us to create the data pipelines for the data platform we are designing now.
-3+ years of hands-on experience in the big-data field: Hadoop, Spark, Spark Streaming, MR.
-3+ years of experience with relevant cloud data services (AWS, Azure, GCP) like EMR, Glue, S3, Lambda, Fargate, DynamoDB, ADF, Azure functions, Azure Blob Storage, DataProc & DataFlow, BigQuery.
-Knowledge of DataBricks will be significant benefit for the candidate.
-Excellent knowledge and hands-on experience of SQL in context of Big Data: Spark SQL, Hive QL
- Ability to understand and optimize Spark execution plans via Spark UI
- Excellent knowledge of Python or Scala
- Batch processing and ETL principles in Data Warehouses.
- Data completeness signals and orchestration
- Approaches for historical reprocessing and data correction
- Handling bad data and late data in inputs and outputs
- Schema migrations and datasets evolution
- B2 Strong speaking English.
- Ability to learn fast new set of tools and technology used internally: Radar, platform services, telemetry providers, Spark-as-a-Service, build system and much more.
Will be a plus
- Understanding of functional programming ideas and principles.
- Experience in building and using web services.
- Experience with any of Teradata, Vertica, Oracle, Tableau.
- Spark Streaming and Kafka.
- Experience or knowledge of: Apache Iceberg, Trino (Presto), Druid, Cassandra, Blob storage like AWS.
- Understanding or experience with Azkaban or Airflow.