Senior Data Engineer with Databricks
Summary
Senior data engineer (8-10 years) who designs, builds, and maintains scalable batch and real-time data pipelines in an enterprise environment. Core stack is Databricks, Apache Spark/PySpark, SQL, and Python, with heavy focus on ETL/ELT, data quality, and performance optimization.
Key responsibilities
Design, develop, and maintain scalable and reliable data pipelines for batch and/or real-time data processing.
Develop and optimize data engineering solutions using Databricks and related cloud data technologies.
Build and maintain data pipelines, ETL/ELT processes, and data transformation workflows.
Work with large and complex datasets to ensure data quality, consistency, availability, and performance.
Develop and optimize SQL and Python code for data processing and transformation.
Implement data engineering solutions using Apache Spark / PySpark.
Collaborate with Data Architects, Data Scientists, Analysts, Application Developers, and business stakeholders to understand data requirements.
Troubleshoot and resolve data pipeline, integration, and performance issues.
Apply best practices for data governance, security, monitoring, and data quality.
Participate in technical design, code reviews, testing, deployment, and production support.
About you
8–10 years of experience in Data Engineering, Data Development, or a related field.
Strong hands-on experience with Databricks in an enterprise data environment.
Strong experience with Apache Spark / PySpark.
Advanced proficiency in SQL and solid experience with Python.
Strong understanding of ETL/ELT processes, data warehousing, and data pipeline development.
Experience working with large-scale datasets and distributed data processing.
Experience with cloud-based data platforms and services.
Strong understanding of data quality, data modeling, and performance optimization.
Experience with source control and software development best practices.
Strong analytical, problem-solving, and troubleshooting skills.