Senior Data Engineer
Summary
Build and maintain Azure + Databricks pipelines that ingest, transform, and deliver trusted datasets for an automotive application using PySpark, Spark SQL, and Azure Data Factory.
Client: automotive industry
Hourly Rate: up to 140 PLN
Location: Wroclaw, Poland
Work arrangement: hybrid, once per week in the office or fully remote - to be determined, full-time
Maintain and evolve the data flows used by the Picto application: Azure + Databricks pipelines (ADF + notebooks) that ingest data from APIs using Ingestion Framework, transform it (PySpark/Spark SQL), and deliver trusted datasets.
Responsibilities:
- Own day-to-day operations of Picto data pipelines (ingest → transform → publish), ensuring reliability, performance and cost efficiency.
- Develop and maintain Databricks notebooks (PySpark/Spark SQL) and ADF pipelines/Triggers; manage Jobs/Workflows and CI/CD.
- Implement data quality checks, monitoring & alerting (SLA/SLO), troubleshoot incidents, and perform root-cause analysis.
- Secure pipelines (Key Vault, identities, secrets) and follow platform standards (Unity Catalog, environments, branching).
- Collaborate with BI Analysts and Architects to align data models and outputs with business needs.
- Document datasets, flows and runbooks; contribute to continuous improvement of the Ingestion Framework.
Requirements:
- Azure Databricks (PySpark, Spark SQL; Unity Catalog; Jobs/Workflows).
- Azure data services: Azure Data Factory, Azure Key Vault, storage (ADLS), fundamentals of networking/identities.
- Python for data engineering (APIs, utilities, tests).
- Azure DevOps (Repos, Pipelines, YAML) and Git-based workflows.
- Experience operating production pipelines (monitoring, alerting, incident handling, cost control).
Nice to have:
- AI
Soft skills:
- Proactive ownership and “driver” mindset—able to move topics forward end-to-end.
- Collaborative and business-oriented; comfortable working with IT and business stakeholders.
- Open-minded, flexible, and quality-focused; clear written documentation and communication.
Tech stack:
- Azure, Databricks (PySpark/Spark SQL, Unity Catalog, Workflows), ADF, ADLS/Delta, Key Vault, Azure DevOps (Repos/Pipelines YAML), Python, SQL
If you meet most of the requirements and are looking for your next challenge, we’d love to hear from you - feel free to apply below!