Senior Data Engineer
Summary
Build and maintain data pipelines using Azure, Databricks, and PySpark, collaborating with stakeholders to deliver clean, scalable data solutions in an Agile environment.
Key responsibilities:
- Collaborate with business stakeholders to understand requirements and translate them into data engineering solutions.
- Design, build, and maintain end-to-end data pipelines for ingestion, transformation, and processing using Azure services, Databricks, IICS, and Azure Synapse.
- Develop advanced transformation logic in Databricks and IICS to support complex business use cases.
- Knowledge of modern Data Modeling and Design concepts
- Set up and manage DevOps environments, including configurations, CI/CD pipelines, releases, deployments, and hotfix management.
- Write clean, efficient, and maintainable code in SQL, Python, PySpark, and scripting languages.
- Actively contribute in an Agile environment, ensuring transparency, collaboration, and continuous improvement.
Required skills:
- Several years of experience in Data Engineering.
- Strong hands-on experience with Databricks, Azure Services (Azure Data Factory, Azure SQL Database, Azure Key Vault), IICS, and Azure Synapse.
- Advanced programming skills in Python, PySpark, and SQL.
- Experience designing and implementing CI/CD pipelines and managing Azure DevOps environments.
- Knowledge of API integrations and enterprise data architecture, theoretical AI knowledge
- Experience working in Agile/Scrum teams.
- Strong analytical and problem-solving skills