Lead Data Engineer

We're seeking a hands-on Lead Data Engineer to construct and maintain scalable data pipelines along with cloud-based data solutions. The right person for this role brings substantial background in Python, PySpark, AWS, Airflow, and DevOps, with particular attention to production-grade engineering and automation practices

Responsibilities

• Create, construct, and sustain scalable data pipelines running on AWS infrastructure
• Build dependable data processing solutions leveraging Python, Pandas, and PySpark
• Establish and oversee workflows via AWS Managed Workflows for Apache Airflow (MWAA)
• Craft and fine-tune data queries through AWS Athena to enable efficient analytical work
• Streamline infrastructure setup and deployment workflows through automation using Terraform and GitHub Actions
• Put in place testing, monitoring, troubleshooting practices, and data-quality safeguards to keep pipelines dependable
• Take part in reviewing code and help promote sound engineering practices throughout the team
• Work alongside engineering, product, and business groups within an Agile setting

Requirements

• At least 5 years of applicable professional background
• A minimum of one year spent leading and managing development teams
• Solid foundation in core Python programming applied to data engineering work
• Practical experience applying Python Pandas for manipulating and analyzing data
• Strong capability with PySpark for handling large-scale distributed data processing
• Background working with AWS MWAA or Apache Airflow to coordinate and manage data workflows
• Familiarity with AWS Athena for running queries and analyzing data housed within AWS
• Hands-on background using Terraform to automate infrastructure setup and management
• Working knowledge of GitHub Actions for constructing automated CI/CD pipelines
• Good grasp of DevOps principles as applied within data engineering contexts
• Strong capabilities in data engineering, troubleshooting, and resolving complex problems
• Upper-intermediate English language skills or higher (B1)

Nice to have

• Background applying GitHub Copilot to assist with development, code reviews, testing, and release processes
• Knowledge of CI/CD methodologies and associated tools for smoother software delivery
• Background working within Agile/Scrum team structures
• Experience gained within financial services or investment management sectors
• Background in Gen AI application development and incorporating Gen AI practices into the software development lifecycle (SDLC)

Benefits

• International projects with top brands
• Work with global teams of highly skilled, diverse peers
• Healthcare benefits
• Employee financial programs
• Paid time off and sick leave
• Upskilling, reskilling and certification courses
• Unlimited access to the LinkedIn Learning library and 22,000+ courses
• Global career opportunities
• Volunteer and community involvement opportunities
• EPAM Employee Groups
• Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn