Associate-Data Engineer

<p><span style="font-size:14px;"><b><u>Job purpose </u></b></span></p><p><span style="font-size:14px;">We are seeking a Data Engineer for a multi-year strategic transformation program modernizing the global </span>middle- and back-office technology platform through a cloud-native, event-driven architecture. The platform leverages AWS, Azure, Kafka-based event streaming, Databricks, modern workflow orchestration, API-led integration and scalable cloud infrastructure. We are seeking a highly motivated Analyst-level Data Engineer to support the design, development and implementation of modern cloud-based data platforms. The candidate will work closely with business stakeholders, solution and data architects, data engineers from other teams at SMBC, and application teams to build scalable data pipelines and streaming solutions that support strategic transformation initiatives. The ideal candidate will have hands-on experience with Databricks, cloud technologies (AWS and Azure) and real-time data processing frameworks; experience working with Apache Kafka, AWS MSK or Confluent Kafka is highly desirable. </p><p><span style="font-size:14px;"><b><u>Key responsibilities </u></b></span></p><p><span style="font-size:14px;">• Develop, maintain and optimize data pipelines using Databricks and cloud-native technologies. </span></p><p><span style="font-size:14px;">• Design and implement batch and real-time data ingestion frameworks. </span></p><p><span style="font-size:14px;">• Build and support streaming data solutions using Databricks Structured Streaming. </span></p><p><span style="font-size:14px;">• Integrate data sources with event-streaming platforms such as AWS MSK (Managed Streaming for </span>Kafka) and Confluent Kafka. </p><p><span style="font-size:14px;">• Develop ETL/ELT processes to transform and load data into enterprise data platforms. </span></p><p><span style="font-size:14px;">• Participate in data modeling, data-quality validation and metadata management activities. </span></p><p><span style="font-size:14px;">• Collaborate with business, technology and architecture teams to understand data requirements and </span>translate them into scalable solutions. </p><p><span style="font-size:14px;">• Support cloud migration and modernization initiatives across AWS and Azure environments. </span></p><p><span style="font-size:14px;">• Monitor, troubleshoot and optimize data pipelines for performance, reliability and scalability. </span></p><p><span style="font-size:14px;">• Participate in code reviews, testing, deployment and production-support activities. </span></p><p><span style="font-size:14px;">• Contribute to establishing engineering standards, best practices and reusable frameworks. </span></p><p><span style="font-size:14px;"><b><u>Key competencies </u></b></span></p><p><span style="font-size:14px;">• Bachelor's degree in Computer Science, Engineering, Information Systems or a related field. </span></p><p><span style="font-size:14px;">• 2–5 years of experience in Data Engineering or related technical roles. </span></p><p><span style="font-size:14px;">• Hands-on experience with Databricks development and administration. </span></p><p><span style="font-size:14px;">• Experience developing data pipelines using PySpark and Spark SQL. </span></p><p><span style="font-size:14px;">• Knowledge of cloud platforms including AWS and/or Microsoft Azure. </span></p><p><span style="font-size:14px;">• Experience working with data lakes, distributed computing and large-scale data processing. </span></p><p><span style="font-size:14px;">• Understanding of real-time data streaming concepts and architectures, with familiarity with </span></p><p><span style="font-size:14px;"><b>Databricks Structured Streaming. </b></span></p><p><span style="font-size:14px;">• Working knowledge of Apache Kafka, AWS MSK or Confluent Kafka. </span></p><p><span style="font-size:14px;">• Experience with source-control systems such as Git, and strong SQL skills. </span></p><p><span style="font-size:14px;">• Strong analytical, problem-solving and communication skills. </span></p><p><span style="font-size:14px;"><b>Preferred qualifications </b></span></p><p><span style="font-size:14px;">• Experience with Delta Lake architecture and Lakehouse implementations. </span></p><p><span style="font-size:14px;">• Knowledge of event-driven architectures and messaging patterns. </span></p><p><span style="font-size:14px;">• Experience with CI/CD pipelines and DevOps practices. </span></p><p><span style="font-size:14px;">• Familiarity with Terraform, CloudFormation or other Infrastructure-as-Code frameworks. </span></p><p><span style="font-size:14px;">• Experience with financial services, capital markets or regulatory reporting platforms. </span></p><p><span style="font-size:14px;">• Understanding of trade lifecycle concepts across FX, Rates, Derivatives and Securities. </span></p><p><span style="font-size:14px;">• Exposure to enterprise integration technologies and API-based architectures. </span></p><p><span style="font-size:14px;">• Additional preferred tooling: Airflow, Docker, Kubernetes, Python, REST APIs and data modeling. </span></p><div><br /></div>