Data SDE 2 - Scala
Posted Updated
Description
Mandatory:
- Strong experience in Data Engineering, including data processing, ingestion from diverse sources and file formats.
- Hands-on experience building and implementing large-scale distributed data platforms and applications.
- Strong hands-on experience with Apache Spark and Scala.
- Advanced Scala programming experience, including Functional Programming, Case Classes, Complex Data Structures and Algorithms.
- Strong understanding of SQL, Spark, Query Tuning and Performance Optimisation.
- Experience building, developing and maintaining large-scale distributed systems and data processing applications.
- Good understanding of software architecture, design principles, SOLID and DRY principles.
- Experience with Big Data processing frameworks such as Hadoop, Spark and Spark Streaming.
- Experience in Big Data Unit, System, Integration and Regression Testing.
- Experience with data validation and data quality.
- DevOps experience with Jenkins, Maven, GitHub, Artifactory/JFrog and CI/CD.
- Experience with code management tools such as Git.
- Experience working with data formats such as ORC, Avro, Parquet, JSON and CSV.
- Experience with Agile processes and tools such as Jira and Confluence.
Good to have:
- Experience with Hadoop and its distributions, such as Cloudera / Hortonworks.
- Experience with data warehousing, data lakes and Medallion Architecture.
- Experience with data streaming technologies such as Kafka and Spark Streaming.
- Experience with Big Data orchestration tools such as NiFi, Airflow, YARN, Oozie and Azkaban.
- Experience with Spark on Kubernetes.
- Experience building and managing large-scale on-premise Hadoop clusters or cloud-based data platforms.
- Experience with a range of Big Data tools, databases and frameworks such as Kafka Connect, Apache Arrow, Apache Iceberg and Flink.
- Experience with SQL and NoSQL databases such as Hive, Cassandra, Dremio and Presto.
- Experience with shell scripting and automation using Ansible or related configuration management tools.
- Exposure to cloud-native storage such as Ceph or S3.
- Exposure to cloud platforms such as Azure, AWS or GCP.
- Experience with Kubernetes, Docker and related container technologies.
What would you do here:
- Design, develop and maintain large-scale Spark applications using Scala.
- Work on complex data processing requirements involving transformations, aggregations and distributed data processing.
- Write efficient, scalable and optimised Spark code.
- Identify and resolve performance bottlenecks in Spark applications.
- Design and implement end-to-end data processing workflows.
- Build and maintain reliable, scalable distributed data systems.
- Follow software architecture and engineering best practices while developing production-grade applications.
- Take ownership of development activities and deliver high-quality solutions independently.
- Work closely with engineering teams and relevant stakeholders to understand requirements and translate them into technical solutions.
- Collaborate with team members across functions and contribute to design discussions and problem-solving.
- Focus on improving existing solutions, automating operational tasks and developing reusable components and frameworks.
- Maintain good documentation and communicate technical decisions effectively.
Skills
- Agile
- Airflow
- Ansible
- Artifactory
- Automation
- Avro
- AWS
- Azure
- Bash
- Big Data
- Cassandra
- Ceph
- CI/CD
- Cloud
- Cloud Native
- Cloudera
- Confluence
- Data Engineering
- Data Lake
- Data Quality
- Data Warehousing
- DevOps
- Distributed Systems
- Docker
- Flink
- GCP
- Git
- GitHub
- Hadoop
- Hive
- Iceberg
- Jenkins
- Jira
- JSON
- Kafka
- Kubernetes
- Maven
- Medallion Architecture
- NiFi
- NoSQL
- Parquet
- S3
- Scala
- Spark
- SQL
Sign in to read and join the discussion.