Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

EPAM Systems

NewBe an early applicant

Lead Data Engineer (PySpark, statistics, MLflow)

Posted Updated

Description

We are looking for a Lead Data Engineer to join our team and take ownership of complex data engineering challenges, from migrating legacy codebases to building robust, production-grade pipelines. In this role, you will work independently on large-scale PySpark systems, apply statistical methods to real-world sensor data, and leverage MLflow to manage the full lifecycle of machine learning models.

Responsibilities

  • Migrate existing pandas, Koalas or pandas-on-Spark code to native PySpark implementations
  • Decompose and refactor large legacy notebooks without relying on existing documentation
  • Apply statistical techniques to detect outliers and events in sensor time series data
  • Package and manage machine learning models using MLflow, including experiment tracking, Model Registry and versioning
  • Implement model explainability techniques such as SHAP for existing models
  • Develop unit and regression tests for PySpark pipelines using pytest and tools like assertDataFrameEqual or chispa
  • Work independently on data engineering projects with minimal oversight
  • Integrate AI coding assistants into daily workflows and critically review their output
  • Utilize Window functions to optimize data transformation logic

Requirements

  • 5+ years of experience in data engineering with deep knowledge of PySpark and pandas
  • Proven experience migrating pandas, Koalas or pandas-on-Spark code to native PySpark, including Window functions
  • Capability to read and decompose large legacy notebooks without documentation, and to work independently on them
  • Expertise in applied statistics, including outlier detection and event detection in sensor time series
  • Proficiency in MLflow for experiment tracking, Model Registry and versioning, with skills in packaging existing models including explainability techniques like SHAP
  • Skills in writing unit and regression tests for PySpark using pytest and assertDataFrameEqual or chispa
  • Familiarity with using AI coding assistants in daily work and reviewing their output critically
  • English proficiency at B2 level or higher

Nice to have

  • Knowledge of incremental processing on Delta, including MERGE and Change Data Feed, and Databricks job triggers
  • Background in working with sensor or IoT data
  • Databricks Machine Learning Professional certification
  • German proficiency

Benefits

We believe that the greatest strength of the company is its people. EPAM is fully committed to help its employees to reach their full potential and achieve their professional goals through continues learning. With this in mind, we would like to introduce to you few of the many opportunities and services which we believe will help you expand your current knowledge:

  • Full access to cutting-edge tools and technologies
  • Competitive compensation depending on experience and skills
  • All-around Social package: professional & soft skills training, medical & family care programs, sports
  • Free English classes
  • Unlimited access to LinkedIn learning solutions
  • Continuous experience exchange with experts and professionals worldwide
  • Friendly team and comfortable working environment
  • Engineering, corporate, and social events within and outside the Company
  • Flexible working schedule
  • Opportunities for self-realization

Skills

What Lead Data Engineering jobs ask for — and how much of it you have →
Apply

See also

Data Engineering jobs by country — openings, pay and top skills →