Azure Data Engineer

Roles and Responsibilities :

• Design, develop, test, deploy and maintain large-scale data pipelines using Azure DataBricks on Azure Databricks platform.
• Collaborate with cross-functional teams to gather requirements and design solutions for complex data processing needs.
• Develop high-quality code in PySpark, SQL, and other relevant technologies to ensure efficient data processing and analysis.
• Troubleshoot issues related to pipeline failures or performance problems.

Job Requirements :

• 5-10 years of experience in designing and developing big data solutions using Hadoop ecosystem (HDFS, YARN, MapReduce).
• Strong expertise in working with Azure Databricks platform including creating clusters, jobs, datasets etc.
• Proficiency in writing efficient Python code using PySpark framework for big data processing tasks.
• Experience with SQL querying languages such as T-SQL.