Spark / Scala Developer (AWS) – Mid-Level
Description
About the role
Sector: Banking
We are looking for a Data Engineer specialized in Apache Spark and Scala, with experience in AWS environments and large-scale data processing, to join a project in the banking sector.
The selected person will participate in the design, development, maintenance, and optimization of ETL processes and data pipelines, working in a production and regulated environment. We are looking for an autonomous, analytical, and results-oriented profile, capable of managing data processes, performing validations, and resolving technical incidents with little supervision.
What you'll do
- Design, develop, and maintain data pipelines using Apache Spark and Scala.
- Implement ETL processes, transformations, and data migrations, guaranteeing their integrity and consistency.
- Orchestrate and monitor batch processes using AWS Step Functions.
- Work with AWS services such as EMR, Glue, and Lambda.
- Supervise production processes, analyze Hadoop/YARN logs, and diagnose errors.
- Perform quality controls and data validations using SQL and Amazon Athena.
- Investigate incidents, identify root causes, and propose corrective solutions.
- Participate in the definition of technical solutions and translate business requirements into specifications.
- Document developments, tests, and operating procedures.
- Collaborate with technical and functional teams.
What we're looking for
Experience: 2-3 years minimum
English: C1 (essential)
- At least 2-3 years of experience in Data Engineering.
- Hands-on experience with Apache Spark, Scala, and Hadoop.
- Solid knowledge of AWS EMR, Glue, Lambda, and Step Functions.
- Experience developing and maintaining ETL processes and data pipelines.
- Knowledge of Terraform for infrastructure configuration.
- Experience in monitoring and resolving incidents in production environments.
- Proficiency in SQL and tools such as Amazon Athena.
- Experience in batch processing and process orchestration.
- Knowledge of Python, Bash, and Unix/Linux environments.
- Experience with Git, Maven, and CI/CD tools such as Jenkins.
- C1 English is essential.
Nice to have
- Previous experience in banking or financial services.
- Knowledge of regulatory reporting.
- Experience with Control-M.
- Knowledge of Power BI.
- Experience in functional analysis, definition of test cases, and technical documentation.
What you'll get
MAX Salary: 32Ks
How you'll work
Location: Preferably in one of the client's offices, following their onsite model. Other locations will be valued.
About the company & team
We are looking for a proactive, autonomous, and decisive person, with the ability to work in production environments, manage incidents, analyze technical problems, and take responsibility for their deliverables. We will especially value the ability to adapt and collaborate with multidisciplinary teams.
Skills
Sign in to read and join the discussion.