Data Engineer

We are seeking a skilled Data Engineer with expertise in Azure Databricks, PySpark, and Python to join our dynamic team in Karnataka. The ideal candidate will have a strong background in data engineering and a passion for optimizing data processing workflows. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that support our analytics and business intelligence initiatives. Key Responsibilities: - Develop and implement data processing solutions using Azure Databricks and PySpark. - Optimize data workflows for performance and efficiency, focusing on Spark optimization techniques and performance tuning. - Collaborate with data scientists and analysts to understand data requirements and deliver high-quality datasets. - Work with Delta Lake to manage data lakes and ensure data integrity and reliability. - Monitor and troubleshoot data pipeline performance, implementing improvements as necessary. - Stay updated with the latest trends and technologies in data engineering and contribute to best practices within the team. Required Skills: - Proficiency in Azure Databricks and PySpark. - Strong programming skills in Python. - Experience with Delta Lake and data lake management. - Knowledge of Spark optimization techniques and performance tuning. - Familiarity with AQE (Adaptive Query Execution) and its application in data processing. - Excellent problem-solving skills and the ability to work collaboratively in a team environment. If you are a motivated Data Engineer looking to make an impact in a fast-paced environment, we encourage you to apply and join our innovative team.