Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Shore

NewBe an early applicant

Big Data Engineer – Spark/Scala

Posted Updated

Summary

Big Data Engineer developing new features in Spark/Scala on an on-premises Cloudera Hadoop platform hosting a Finance and Risk data lake, in an Agile/DevOps context. Core stack: Spark 3, Scala 2.12, HDFS/Hive/HBase/Kafka secured with Kerberos/Ranger, plus Jenkins/XL Deploy CI/CD and Control-M scheduling.

Role overview

We are looking for a Big Data Engineer to strengthen projects built on an on-premises Cloudera Hadoop Big Data platform that hosts a Finance and Risk data lake. The role focuses on designing and developing new features in Spark/Scala within an Agile and DevOps context, with a strong emphasis on code quality, reuse, and scalable architecture.

Key responsibilities

  • Take part in the technical design of the features to be developed.
  • Develop new features in Spark/Scala.
  • Rationalise and reuse components to build an optimised, non-redundant, consistent and scalable architecture.
  • Take part in code reviews and help improve overall code quality.
  • Take part in deployments to test and production environments in a DevOps context.
  • Manage component upgrades and the related testing.
  • Write and maintain technical documentation.
  • Join the team’s Agile ceremonies and use tools such as Jira and Confluence.

Technical environment

  • Spark 3, ideally 3.2
  • Scala 2, ideally 2.12
  • Cloudera Hadoop platform
  • YARN, HDFS, Hive, HBase, Kafka
  • Kerberos, Ranger
  • Jenkins, XL Deploy, XL Release
  • Control-M
  • Bitbucket/Git, Jira, Confluence
  • IntelliJ, VS Code
  • ZIO, Cats

  • 3 to 6 years of experience in data engineering.
  • Proficiency in Spark 3 (ideally 3.2) and Spark application optimisation techniques.
  • Proficiency in Scala 2 (ideally 2.12) used with Spark, and in coding best practices.
  • Very good knowledge of the Hadoop ecosystem (Cloudera, YARN, HDFS, Hive, HBase, Kafka).
  • Knowledge of security components such as Kerberos and Ranger.
  • Experience implementing CI/CD pipelines (e.g. Jenkins, XL Deploy, XL Release).
  • Experience with Control-M scheduling and the Atlassian suite (Bitbucket/Git, Jira, Confluence).
  • Familiarity with IntelliJ or VS Code.
  • Knowledge of a functional programming framework such as ZIO or Cats is an advantage.
  • Ability to understand technical requirements and to plan and estimate tasks.
  • Experience working in Agile (Scrum) with a continuous-improvement mindset.
  • English: B2 minimum, C1 preferred.
  • Ability to adapt communication to infrastructure teams, developers, business analysts and users.

Candidate-se aqui:

Skills

Apply

See also

Data Engineering jobs by country — openings, pay and top skills →