Data Engineer (Azure Databricks, PySpark, Delta Lake, BigQuery, Data Observability, JMeter)

Responsibilities:

• Develop and maintain reliable data pipelines for business and analytical applications.
• Use Azure Data Factory and Azure Databricks to manage data ingestion, transformation, and processing workflows.
• Write optimized SQL, Python, and PySpark code for data preparation and processing.
• Work with Delta Lake and Google BigQuery for storing and managing analytical data.
• Monitor data processes and identify data-related issues using data observability practices.
• Build and maintain REST API integrations for exchanging data between applications and services.
• Process and validate JSON data used within data and API workflows.
• Support automated build and deployment activities through Azure DevOps.
• Develop Power BI dashboards and reports to present data-driven insights.
• Perform API and application performance testing using JMeter to assess response time, reliability, and system performance.
• Investigate technical issues, improve processing efficiency, and coordinate with relevant teams to resolve data-related challenges.

Requirements:

• Qualification in Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related technical discipline.
• 4+ years of professional experience in Data Engineering, Data Analytics, or a related technical role.
• Strong Hands - on experience with SQL and Azure Databricks for data processing and analytical workloads.
• Strong Experience in PySpark, Python, and Azure Data Factory for developing and managing data processing workflows.
• Experience in practical knowledge of Azure DevOps, Delta Lake, REST API, JSON, and Google BigQuery.
• Experience in working with cloud-based data platforms and developing scalable data pipelines.
• Strong Experience of Data Observability, data monitoring, and data reliability practices.
• Experience in developing Power BI dashboards and reports for business and analytical requirements.
• Hands - on experience in JMeter for API, load, or performance testing.
• Hands - on experience in ETL.
• Experience in Anomaly.
• Strong analytical, troubleshooting, and problem-solving skills.
• Good communication and collaboration skills with the ability to work effectively with cross-functional teams.