Spark SCALA Developer
Posted Updated
Description
Job Description
We are seeking a highly skilled Spark Scala Engineer with 5-7 years of experience in designing, developing, and maintaining large-scale data processing applications. The ideal candidate should have strong expertise in Apache Spark, Scala, AWS services, and hands-on experience working on data migration projects. Experience with MongoDB schema validation, cloud-native architectures, and data pipeline optimization is highly desirable.
Roles & Responsibilities
Key Responsibilities
- Design, develop, and optimize scalable batch and real-time data processing solutions using Apache Spark and Scala.
- Build and maintain robust data ingestion frameworks for processing high-volume data from multiple source systems.
- Participate in large-scale data migration projects, including data analysis, transformation, validation, and reconciliation activities.
- Develop and manage ETL/ELT pipelines using AWS services such as EMR, S3, and Step Functions.
- Analyze and resolve data quality issues, ensure data consistency, and support production troubleshooting.
- Work closely with business stakeholders, data architects, and cross-functional teams to gather and translate requirements into technical solutions.
- Implement monitoring, logging, and performance tuning of Spark applications.
- Perform schema design, validation, and data modeling activities in MongoDB.
- Conduct root cause analysis for data ingestion and processing issues and implement preventive measures.
- Ensure adherence to coding standards, security guidelines, and best practices.
- 3-6 years of hands-on experience in Scala and Apache Spark development.
- Strong experience in building ETL/Data Ingestion pipelines.
- Extensive knowledge of AWS Cloud Services, including:
- Amazon EMR
- Amazon S3
- AWS Step Functions
- IAM and CloudWatch (preferred)
- Experience working on data migration projects involving legacy-to-cloud or platform modernization initiatives.
- Strong SQL skills and experience with large-scale data processing.
- Hands-on experience with MongoDB, including:
- Schema design
- Schema validation
- Data modeling
- Query optimization
- Experience in handling structured and semi-structured datasets (JSON, CSV, Avro, Parquet, etc.).
- Familiarity with CI/CD pipelines, Git, and Agile development methodologies.
- l scripting for automating tasks.
Skills
Sign in to read and join the discussion.