Data Engineer
Summary
Build and maintain ETL pipelines, workflow orchestration, and cloud-based data platforms to move and process financial data reliably and at scale.
Key Responsibilities
Integration & Workflow Engineering
- Analyse and re-platform existing workflow-based integrations from one platform to another while maintaining functional parity.
- Implement orchestration logic including triggers, conditional routing, retries, exception handling, and state management.
- Design and implement automation patterns to handle platform limitations (e.g., looping, batching, pagination, throttling).
- Ensure workflows are idempotent, fault tolerant.
- Support environment-based deployments (DEV / UAT / PROD) with configuration-driven designs.
- Design and build ETL pipelines for ingesting flat files (e.g., CSV) into relational databases.
- Handle schema validation, basic schema evolution, data quality checks, and error reconciliation.
- Optimize data ingestion for performance, scalability, and reliability.
- Collaborate with application teams to understand upstream and downstream data dependencies.
- Build and maintain data pipelines on cloud-based big data platforms using distributed processing frameworks.
- Contribute to Lakehouse-style data storage that supports both batch and streaming data.
- Work with modern table formats that support incremental processing, versioning, and historical queries.
- Support use cases such as append-heavy datasets, high-write event data, and analytical queries.
- Implement logging, monitoring, and alerting for workflows and data pipelines.
- Support operational readiness including runbooks, deployment procedures, and rollback strategies.
- Participate in root-cause analysis and continuous improvement of data pipelines.
- Ensure adherence to data governance, security, and compliance standards.
- Strong experience with workflow orchestration / integration platforms.
- Solid understanding of ETL concepts and hands-on experience with file-based ingestion.
- Proficient in SQL and working knowledge of relational databases (e.g., SQL Server or equivalent).
- Experience with distributed data processing frameworks (e.g., Spark).
- Familiarity with streaming and batch data processing concepts.
- Practical experience with cloud platforms and managed data services.
- Understanding of CI/CD principles for data and integration workflows.
- Experience with scripting or programming languages commonly used in data engineering (e.g., Python).
- Ability to build reusable utilities for batching, retries, pagination, and error handling.
- Experience with REST APIs and structured data formats (JSON, CSV).
- Exposure to modern Lakehouse table formats supporting incremental processing and time travel.
- Experience with stream processing engines (e.g., Flink or Spark Streaming).
- Familiarity with query engines used for analytical access.
- Experience working in regulated environments (banking, financial services, or manufacturing).
- Knowledge of data observability and quality frameworks.
- Typically, 3–5 years of relevant experience in data engineering, integration engineering, or similar roles.
- Proven ability to work independently on moderately complex problems while collaborating within a larger team.
- Strong analytical and problem-solving skills.
- Ability to understand existing systems and translate requirements into working solutions.
- Clear communication with technical and non-technical stakeholders.
- Detail-oriented with a focus on reliability and maintainability.
- Comfortable working across multiple teams and applications.