Big Data Engineer - ETL/Apache Spark

Role Overview :

We are looking for an experienced Big Data Engineer with strong hands-on expertise in Apache Spark and Scala to design, develop, and optimize large-scale data processing solutions. The role will involve building scalable data pipelines, processing high-volume datasets, optimizing distributed workloads, and collaborating with data and technology teams to deliver reliable data solutions.

Key Responsibilities :

- Design, develop, and maintain scalable big data processing applications using Apache Spark and Scala.

- Build data processing and transformation pipelines for large and complex datasets.

- Develop efficient distributed data-processing solutions with a strong focus on performance, scalability, and reliability.

- Optimize Spark jobs, transformations, queries, memory utilization, and cluster workloads.

- Analyze data-processing requirements and translate them into scalable technical solutions.

- Work with structured and unstructured data and implement appropriate processing and transformation techniques.

- Troubleshoot data pipeline failures, performance bottlenecks, and distributed processing issues.

- Develop reusable components and follow software engineering and data engineering best practices.

- Collaborate with Data Engineers, Architects, Analysts, and business stakeholders to understand requirements and deliver data solutions.

- Participate in code reviews, testing, deployment, and production support activities.

- Maintain technical documentation covering data pipelines, processing logic, and operational procedures.

- Contribute to continuous improvement of data processing frameworks, performance, and engineering practices.

Required Skills & Experience :

- 6 - 11 years of experience in Big Data, Data Engineering, or a related technology domain.

- Strong hands-on expertise in Apache Spark and Scala.

- Strong understanding of Big Data concepts, distributed computing, and parallel data processing.

- Experience developing large-scale data processing, transformation, and ingestion solutions.

- Strong understanding of Spark architecture, execution models, partitioning, transformations, actions, and performance optimization.

- Experience troubleshooting and tuning distributed data-processing workloads.

- Good knowledge of SQL and relational data concepts.

- Strong analytical and problem-solving skills.

- Good understanding of software development lifecycle, testing, version control, and deployment practices.

- Ability to work effectively in a collaborative, fast-paced engineering environment.

- Strong communication and stakeholder-management skills.