Data Engineer with PySpark
Summary
Build and maintain scalable cloud-based data pipelines using Python, PySpark, and AWS to support reliable, high-performance data products and systems.
Job responsibilities
- Deliver end-to-end data pipeline solutions on cloud infrastructure leveraging the latest technologies and the best industry practices.
- Use domain modelling techniques to allow us to build best in class business products.
- Structure software so that it is easy to understand, test and evolve.
- Build solutions that avoid single points of failure, using scalable architectural patterns.
- Develop secure code so that our customers and ourselves are protected from malicious actors.
- Promptly investigate and fix issues and ensure they do not resurface in the future.
- Make sure our releases happen with zero downtime for our end-users.
- See that our data is written and read in a way that's optimized for our needs.
- Keep an eye on performance, making sure we use the right approach to identify and solve problems.
- Ensure our systems are reliable and easy to operate.
- Keep us up to date by continuously updating our technologies and patterns.
- Support the products you've built through their entire lifecycle, including in production and during incident management
Required qualifications, capabilities & skills
- Formal training or certification on data engineering concepts and applied experience
- Recent hands-on professional experience as a data engineer
- Experience in coding in a recent version of the Python programming language
- Experience in designing and implementing effective tests (unit, component, integration, end-to-end, performance, etc.)
- Excellent written and verbal communication skills in English
- Experience with AWS cloud technologies and distributed systems
- Experience with data transformation frameworks and data pipeline orchestration tools
- Experience with managing large volumes of data and optimizing data processing
- Understanding of event-base architecture, data streaming and messaging frameworks
- Coach other team members on coding practices, design principles, and implementation patterns that lead to high-quality maintainable solutions.
- Manage stakeholders and effectively prioritize work across multiple work streams.
Preferred qualifications, capabilities & skills
- Experience in working in a highly regulated environment / industry
- Experience with AWS cloud technologies
- Experience with data governance frameworks
- Understanding of incremental data processing and versioning
- Understanding of RESTful APIs and web technologies