Data Engineer (Azure)
Summary
Design, develop, and optimize scalable Azure data platforms and pipelines (Data Lake Gen2, Blob Storage, Spark, SQL) supporting analytics and BI across the organization.
We are looking for a Data Engineer III to join our Data Engineering team. In this role, you will design, develop, and optimize scalable data platforms and pipelines on Microsoft Azure, enabling analytics, reporting, and business intelligence across the organization.
Key Responsibilities
- Utilize Azure data storage accounts for organizing and maintaining data pipeline outputs. (e.g., Azure Data Lake Storage Gen 2 & Azure Blob storage).
- Collaborate with data scientists, data analysts, data architects and other stakeholders to understand data requirements and deliver high-quality data solutions.
- Optimize data pipelines in the Azure environment for performance, scalability, and reliability.
- Ensure data quality and integrity through data validation techniques and frameworks.
- Develop and maintain documentation for data processes, configurations, and best practices.
- Monitor and troubleshoot data pipeline issues to ensure timely resolution.
- Stay current with industry trends and emerging technologies to ensure our data solutions remain cutting-edge.
- Manage the CI/CD process for deploying and maintaining data solutions.
- Bachelor’s degree in computer science, Engineering, or a related field (or equivalent experience) and able to demonstrate high proficiency in programming fundamentals.
- Proven experience (7+ years) as a Data Engineer or similar role dealing with data and ETL processes.
- Experience utilizing SQL DML to query modern RDBMS in an efficient manner (e.g., SQL Server, PostgreSQL).
- Strong understanding of Software Engineering principles and how they apply to Data Engineering (e.g., CI/CD, version control, testing).
- Experience with big data technologies (e.g., Spark).
- Strong problem-solving skills and attention to detail.
- Excellent communication and collaboration skills