Big Data Engineer (Spark/Scala/Java)_WROCLAW OR KRAKOW
Summary
Build and maintain a scalable big-data platform on Azure Databricks using Spark, Scala, and Java, enabling teams to run analytics and regulatory workloads.
Hybrid -> 3 days per week from the client's office
Project Description
ACQA is built on Microsoft Azure cloud computing technology. It aims to deliver:
- Scalable cost-efficient infrastructure, using cloud PaaS components.
- Single core platform, open architecture, designed for change, itemised $cost metrics, automated data lineage.
- Shared across Front Office, Finance and Risk, improving regulatory compliance.
- One-Platform / One-Experience -fast to train, easy to operate, retaining talent.
Responsibilities
- Work with multiple federated teams to enable them to build Big Data solutions
- Manage the ACQA Databricks platform – provide monitoring and other tooling
- Provide best practices for runtime upgrades
Mandatory Skills Description
- Development of big data technologies like Apache Spark and Azure Databricks
- Programming complex production systems in Java and Scala
- Experience in a platform engineering role on a major cloud provider (ideally Microsoft Azure)
- Development and/or operations of CI/CD pipelines using modern cloud-friendly build systems like GitLab
- Experience with containerisation in development, build, and runtime environments, including experience with Kubernetes
Nice-to-Have Skills Description
- Production experience with Terraform on a major public cloud provider (ideally Microsoft Azure)
- Testing and instrumentation practices, observability and alerting tooling
- Understanding of information modelling, data structures and algorithms
- Technical architecture and low-level design
- Agile development and planning practices
- Ability to collaborate effectively in a large global team and influence key architectural decisions