Principal Data Engineer
Description
Duration: Long Term Contract
Job Summary:
-
We are seeking a visionary Principal Data Engineer to lead the delivery of innovative data analytics solutions, with a focus on technical excellence, governance, and operational performance in a high-volume data environment. This role involves strategic and technical leadership in building Azure-based data lake solutions, driving maturity in data management, and engaging with cross-functional teams to leverage data for organizational success.
Key Responsibilities:
-
Architect, design, and implement scalable Azure-based data lake solutions.
-
Establish roadmaps for capability-building across technologies such as Qlik Replicate, Databricks Delta Live Tables, and Unity Catalog.
-
Provide technical leadership, ensuring adherence to best practices and quality standards.
-
Lead the development of data transformation pipelines using Databricks Delta Live Tables and Structured Streaming.
-
Optimize Databricks resources for high-volume data processing without exceeding budget.
-
Oversee the use of Databricks Unity Catalog for data governance, lineage, and metadata management.
-
Develop and enforce robust data governance policies, ensuring compliance, security, and data quality.
-
Monitor data pipeline performance and proactively resolve production issues.
-
Collaborate with cross-functional teams to align project objectives with business goals.
-
Translate technical challenges into actionable insights for executive teams.
-
Mentor and develop a high-performing team skilled in Azure, Databricks, and data streaming technologies.
-
Facilitate knowledge transfer sessions to empower internal teams with cutting-edge methodologies.
Education and Experience:
-
Bachelor's or Master's degree in IT or a related discipline.
-
15-17 years of experience in the IT industry with expertise in delivering cost-effective cloud-based data analytics solutions.
-
Proven track record in building data lake architectures and optimizing performance in big data environments.
-
Certification in Azure Data Engineering or Databricks (preferred).
Technical Skills:
-
Strong knowledge of Azure technologies, including ADF, ADLS, Databricks, Stream Analytics, Delta Live Tables, and Synapse Analytics.
-
Advanced proficiency in SQL and programming languages such as Python, Scala, or Spark-SQL.
-
Experience in designing and developing scalable ETL pipelines and optimizing data workflows.
-
Deep understanding of data migration from on-premises to cloud, cloud to cloud.
Leadership and Soft Skills:
-
Proven ability to lead diverse teams and manage end-to-end analytics projects.
-
Exceptional communication and collaboration skills for engaging executives and stakeholders.
-
Strong sense of ownership, accountability, adaptability, and commitment to results.
-
Expertise in identifying risks and implementing effective mitigation strategies.
Nice to Have:
-
Familiarity with big data tools like Hadoop, Kafka, and Spark.
-
Experience with visualization tools such as PowerBI or QlikSense.
-
Knowledge of AI/ML, NLP, and GenAI concepts.
-
Familiarity with AGILE and KANBAN methodologies.
Thanks & Regards.
Skills
Sign in to read and join the discussion.