Junior Data Engineer: Scalable Batch & Streaming Pipelines
Posted Updated
Description
Beyondsoft International offers a data engineering role focused on building and maintaining batch and streaming data pipelines. You will map, transform and validate data, support scheduled workflows and monitor pipelines, and help deploy workloads in containerised environments.
Join a team that values data quality, Agile practices and modern tech like Spark, Kafka and Airflow to enable scalable data solutions across markets.
Develop and maintain batch and streaming data pipelines. Perform data mapping, transformation and validation. Support scheduled data workflows and pipeline monitoring. Assist in deploying and maintaining data workloads in containerised environments. Troubleshoot data pipeline issues and support defect resolution. Work with the team to maintain data quality and support CI/CD processes. Minimum 2 years of experience in data engineering or backend development. Familiarity with Agile development methodologies. Working knowledge of SQL and Java or Scala. Basic experience with Apache Spark, Apache Flink or Apache Kafka. Familiarity with Kubernetes, Docker and S3-compatible object storage. Experience with data mapping, transformation, validation and scheduled workflows. Familiarity with the SAFe framework. Exposure to MinIO or equivalent S3-compatible storage solutions. Familiarity with relational databases, Git, CI/CD and automated testing. Exposure to Apache Airflow or equivalent workflow orchestration tools. Knowledge of batch and stream-processing technologies.Skills
Sign in to read and join the discussion.