Databricks Engineer
Posted
Summary
Builds and maintains large-scale data processing pipelines with PySpark on AWS EMR — designing, testing, deploying pipelines, troubleshooting Spark/ETL issues, and writing technical documentation. Requires 5-8 years of big data development experience; Databricks proficiency is a plus.
Roles and Responsibilities :
- Design, develop, test, deploy and maintain large-scale data processing pipelines using PySpark on AWS EMR.
- Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions that meet business needs.
- Troubleshoot complex issues related to data processing, Spark jobs, and ETL processes.
- Develop technical documentation for the developed solutions.
Job Requirements :
- 5-8 years of experience in software development with a focus on big data technologies such as Hadoop, Spark, etc.
- Strong proficiency in Python programming language and its libraries like NumPy, Pandas, Matplotlib.
- Experience working with cloud-based platforms like AWS EMR (Elastic Map Reduce) is required.
- Proficiency in Data Bricks is an added advantage.