Data Engineer Specialist- ScotiaTech

Purpose

The Data Engineer Specialist designs, develops, and maintains data pipelines and infrastructure on Microsoft Azure and Azure Databricks to support the creation of scalable data products for International Banking. Reporting to the Senior Data Engineer, this role focuses on implementing robust, efficient, and secure lakehouse-based data solutions that enable data-driven decision-making and support business objectives across International Banking markets, while contributing to the migration and rationalization of legacy workloads onto the Azure Databricks target platform.

Accountabilities

• Build, test, and maintain data pipelines and ETL/ELT processes on Azure and Azure Databricks (Delta Lake, medallion architecture) to ensure reliable and efficient data flow for data products.
• Collaborate with the Senior Data Engineer and cross-functional teams (Data Scientists, Product Owners, Architecture, and Platform Engineering) to understand requirements and deliver high-quality data solutions aligned with international business goals.
• Implement data models, schemas, and transformations (dimensional and Data Vault patterns) to support analytics and reporting needs.
• Ensure data quality, integrity, and performance by monitoring and optimizing data pipelines, including performance tuning and cost optimization (FinOps) of Spark workloads.
• Adhere to and help apply data governance, security, and compliance standards within Azure environments, including Unity Catalog, role-based access control, data lineage, encryption, and key management.
• Troubleshoot and resolve issues in data pipelines to minimize downtime and ensure operational efficiency, performing root-cause analysis where needed.
• Contribute to the adoption of best practices and tools for data engineering, including documentation, automated testing, and DevOps/DataOps practices.
• Support the migration and rationalization of legacy workloads (including existing Google Cloud Platform estates) onto the Azure Databricks target platform, minimizing business and regulatory risk.
• Stay updated on modern Azure and Databricks data engineering trends and tools to enhance pipeline capabilities.

Dimensions

Colombia, Canada

Business Unit: IBTT

Education / Experience / Other Information

Education

Bachelor’s degree in Computer Science, Data Engineering, Information Technology, Software Engineering, or a related field.

A Master’s degree is considered a plus.

Relevant cloud or data engineering certifications (Databricks, GCP, Azure, AWS) are considered a plus.

Experience

3-5 years of experience in Data Engineering, designing and developing scalable data solutions in cloud environments.

Hands-on experience with Azure Databricks, including data pipeline development, Delta Lake, Spark, notebooks, workflows, Unity Catalog, and Lakehouse architecture.

Strong proficiency in Apache Spark (PySpark) and Python for large-scale data processing.

Experience building batch and streaming data pipelines using modern data engineering practices.

Knowledge of data migration, cloud modernization, and platform transformation initiatives is highly desirable.

Strong SQL skills and experience working with large-scale analytical databases and data warehouses.

Experience with data modeling, schema design, and data warehousing concepts.

Familiarity with DevOps and DataOps practices, including Git-based version control, CI/CD pipelines, automated testing, and deployment processes.

Understanding data governance, security, monitoring, and observability practices within cloud environments.

Experience working in Agile delivery environments and collaborating with cross-functional teams.

Strong problem-solving skills and ability to troubleshoot complex data processing challenges.

Preferred Qualifications

Databricks certifications (Data Engineer Associate/Professional) are highly desirable; Azure DP-203 is an asset.

Experience with Lakehouse architecture and Delta Lake implementation.

Familiarity with infrastructure-as-code tools such as Terraform or Bicep.

Exposure to data quality, data observability, and monitoring frameworks.

Experience supporting cloud migration initiatives from on-premises or legacy platforms.

Soft Skills

Strong stakeholder management and communication skills.

Excellent analytical thinking and problem-solving abilities.

Adaptability and willingness to learn new technologies in a rapidly evolving data ecosystem.

Ability to work effectively in collaborative, distributed, and multicultural teams.

Proactive mindset with a focus on continuous improvement and operational excellence.

Strong ownership and accountability for deliverables and outcomes.

Working Conditions

Work in a standard office-based environment , with standard hours .