Senior Data Engineer - Azure Databricks/Snowflake
Summary
Senior data engineer designing and maintaining large-scale data pipelines on Azure Databricks and Snowflake, building optimized ETL/ELT workflows with Azure Data Factory and Data Lake Storage using Python, PySpark and SQL, and integrating GenAI/LLM capabilities into data workflows.
Key Responsibilities:
- Design, develop and maintain scalable data pipelines using Azure Databricks and Snowflake.
- Build and optimize ETL/ELT workflows for large-scale data processing.
- Work with Azure Data Factory, Azure Data Lake Storage and related Azure services.
- Develop efficient data models and optimize SQL queries for performance.
- Integrate GenAI and LLM capabilities into data engineering workflows.
- Collaborate with data scientists, AI engineers and business stakeholders.
- Ensure data quality, security, governance and reliability.
- Monitor pipeline performance and troubleshoot data processing issues.
Required Skills:
- 6 - 8 years of experience in Data Engineering.
- Strong hands-on experience with Azure Databricks and Snowflake.
- Proficiency in Python, PySpark and SQL.
- Experience developing ETL/ELT pipelines and working with cloud data platforms.
- Knowledge of Azure Data Factory and Azure Data Lake Storage.
- Understanding of data warehousing, data modelling and performance optimization.
- Exposure to Generative AI, LLMs, RAG or AI-integrated data workflows.
- Strong analytical, problem-solving and communication skills.