Data Engineer

We are seeking a skilled Data Engineer to join our dynamic team in Karnataka. The ideal candidate will have a strong background in data engineering, particularly with Azure Databricks and Apache Spark technologies. As a Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines that support our data analytics and business intelligence initiatives. Key Responsibilities: - Develop and implement ETL processes to extract, transform, and load data from various sources into data warehouses and data lakes. - Utilize Azure Databricks and Apache Spark to process large datasets efficiently, ensuring optimal performance and scalability. - Collaborate with data scientists and analysts to understand data requirements and deliver high-quality data solutions. - Optimize Spark jobs for performance, including techniques such as data skew management, broadcast joins, and caching strategies. - Implement data governance practices, including role-based access control (RBAC) and data security measures. - Monitor and troubleshoot data pipeline performance, making necessary adjustments to improve efficiency. - Stay updated with the latest trends and technologies in data engineering and contribute to continuous improvement initiatives. Required Skills: - Proficiency in Azure Databricks and Apache Spark. - Strong programming skills in Python and experience with PySpark. - Solid understanding of SQL and Spark SQL for data manipulation and querying. - Familiarity with Delta Lake and its capabilities for managing data lakes. - Knowledge of Spark architecture and performance tuning techniques. - Experience with data governance and security practices, including RBAC. - Understanding of Azure Data Lake Storage (ADLS Gen2) and its integration with data processing workflows. If you are passionate about data engineering and eager to work in a collaborative environment, we encourage you to apply for this exciting opportunity.