Data Engineer
Summary
Designs and builds scalable data pipelines using Hadoop, Spark, and Kafka to process and integrate large datasets for analytics and ML workloads.
Requirements
- Bachelor's degree in Computer Science or a related field.
- At least 7 years of IT experience of which at least 5 years' experience in similar role, ideally in a Big Data tools
- Proficient understanding of distributed computing principles
- Strong experience of Hadoop, Spark, Iceberg and Spring Boot
- Experience with building stream-processing systems, using solutions such as Spark-Streaming
- Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
- Experience with integration of data from multiple data sources
- Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
- Knowledge of various ETL techniques and frameworks
- Experience with various messaging systems, such as Kafka or RabbitMQ
- Knowlege with Big Data ML toolkits, such as Mahout, SparkML, or H2O
- Good understanding of Lambda Architecture, along with its advantages and drawbacks