Data Engineer (DATABRICKS / PYSPARK)
Posted Updated
Summary
Data engineer building and maintaining batch and near-real-time data pipelines using Databricks, PySpark, SQL, and Delta Lake, implementing Data Lakehouse solutions with Medallion architecture, ETL automation, and pipeline monitoring for Sigma Software, an outsourcing firm based in Warsaw (remote-friendly).
- Develop and maintain data pipelines for processing large volumes of data in batch and near real-time environments
- Work with Databricks, PySpark, and SQL to transform, validate, and optimize data
- Support implementation of Data Lakehouse solutions following engineering standards and best practices
- Build, test, and automate ETL processes and data workflows
- Improve data quality, organization, and reliability across platforms
- Collaborate with engineers, analysts, and stakeholders to deliver scalable solutions
- Troubleshoot, optimize, and monitor data pipelines
- Stay up to date with modern data engineering technologies
- Ensure deliverables meet quality standards and project requirements
- 2+ years of experience in Big Data development and database design
- Strong hands-on experience with SQL
- Strong Python development skills
- Hands-on experience with PySpark
- Experience developing and maintaining data pipelines in Databricks
- Experience with Delta Lake
- Experience with Medallion Architecture (Bronze, Silver, Gold layers)
- Strong analytical and problem-solving skills
- Intermediate or higher English level
WILL BE A PLUS
- Experience with AWS services, including S3