Lead Pyspark Developer
Posted Updated
Description
Luxoft in Warsaw is seeking a senior data engineering lead to develop and optimize data-intensive PySpark applications. You will guide a development team, design scalable data pipelines, and ensure high-quality code and documentation.
The role requires strong Python expertise, experience with Pandas/Numpy, and knowledge of Spark architecture and performance tuning. Collaboration with cross-functional teams and adherence to standards are essential.
Develop software applications based on business requirements. Maintain software applications and implement enhancements per project specs. Participate in requirement analysis, design, development, testing, and deployment activities. Propose new techniques and technologies for software development. Perform unit testing and user acceptance testing to verify functionality. Complete assigned development tasks within deadlines. Adhere to coding standards and best practices. Provide guidance to junior developers when needed. Perform code reviews and recommend improvements. Review requirements and suggest changes to build reliable applications. Develop coding documentation and other technical specifications for projects. Act as a primary contact for development queries and concerns. Analyze and resolve development issues accurately. 10+ years of experience in data intensive PySpark development with Spark architecture and optimization techniques. Experience leading a development team. Proven Python developer with strong Pandas/Numpy dataframes proficiency. Good understanding of Gen AI and data-driven applications. Strong DB connectivity, data loading, transformation and calculation skills. Experience with Pandas/Numpy dataframes, slicing, wrangling and aggregations. Familiarity with vector operations on dataframes/series and apply/map usage. Experience with concurrency and error handling in data pipelines (1–10 GB batches). Ability to translate business requirements into technical specs and design scalable data pipelines. Knowledge of Git and service-oriented architecture with potential integrations.Skills
Sign in to read and join the discussion.