Senior Data Engineer Lead Scotiatech

The Senior Data Engineer is responsible for designing, building, and maintaining scalable data products and pipelines using Microsoft Azure and Azure Databricks, supporting key International Banking initiatives. This role combines deep technical expertise with team leadership, driving the delivery of secure, high-quality lakehouse solutions that enable data-driven decision-making across multiple markets.

As the technical lead, you will own end-to-end solution design and implementation, establish engineering best practices, mentor a team of Data Engineers, and ensure compliance with enterprise architecture, security, and regulatory standards.

Key Responsibilities

• Design, develop, and optimize data pipelines and lakehouse solutions using Azure Databricks, Delta Lake, and Medallion Architecture.
• Lead the technical design and delivery of data engineering initiatives from concept to production.
• Mentor and support Data Engineers through coaching, code reviews, and technical guidance.
• Collaborate with Product Owners, Data Scientists, Architects, and Engineering teams to build scalable data products.
• Implement and maintain best practices for data governance, security, compliance, and access management.
• Monitor data platforms to ensure reliability, performance, data quality, and cost efficiency.
• Drive adoption of modern data engineering practices, including ETL/ELT, DataOps, DevOps, CI/CD, and cloud-native solutions.
• Troubleshoot complex data platform issues and implement long-term solutions.
• Support cloud migration initiatives, including the transition of legacy and GCP-based workloads to Azure Databricks.
• Evaluate emerging technologies and recommend improvements to strengthen data capabilities.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Data Science, or a related field.
• 5+ years of hands-on experience in Data Engineering, with at least 3 years working with Azure and/or Azure Databricks.
• Proven experience leading technical projects and mentoring engineering teams.
• Strong expertise in Azure Databricks, Spark, Delta Lake, Unity Catalog, Workflows, Azure Data Factory, Azure Data Lake Storage Gen2, Event Hubs, and Azure Synapse.
• Advanced proficiency in Python, PySpark, and SQL.
• Experience with data modeling, data warehousing, and lakehouse architectures.
• Knowledge of CI/CD pipelines, Git, Infrastructure as Code (Terraform or Bicep), and automated testing.
• Understanding of cloud security, governance, RBAC, data privacy, and compliance frameworks.
• Experience with cloud migration projects is highly valued.
• Strong English communication skills (B2+ or higher).

Preferred Qualifications

• Azure or Databricks certifications (e.g., DP-203, Databricks Certified Data Engineer Professional).
• Experience in financial services or banking environments.
• Spanish language skills are considered an asset.

Soft Skills

• Strong leadership and stakeholder management capabilities.
• Excellent communication and collaboration skills.
• Results-driven mindset with a strong sense of ownership and accountability.
• Ability to manage multiple priorities in fast-paced environments.
• Proactive problem solver with a continuous improvement mindset.
• Comfortable working with distributed and multicultural teams.