Data Engineer (6 months, Bank) #ESY
Summary
Data Engineer builds and migrates SQL-based reporting pipelines for a bank, replacing legacy data marts with Hive/Impala and validating accuracy.
Data Analysis & Discovery
- Analyse existing tables, reports, datasets and data processing workflows from legacy data marts and manual processes.
- Understand existing scripts, transformation logic, business rules and reporting methodologies.
- Identify data lineage, dependencies and reporting requirements.
- Work closely with business users to understand current reporting processes and migration requirements.
Data Migration & Source Mapping
- Perform source to target mapping between legacy systems and new enterprise data marts.
- Document field mappings, transformation logic and business rules.
- Identify data gaps and recommend improvements where appropriate.
- Develop migration specifications aligned with business requirements.
Development
- Develop and enhance SQL scripts for data extraction, transformation and reporting.
- Work with Hive and Impala to support data migration activities.
- Rebuild existing reporting datasets using the new enterprise data marts.
- Support workflow automation where applicable.
Testing & Validation
- Perform reconciliation between legacy and migrated datasets.
- Validate data accuracy, completeness and consistency.
- Investigate and resolve data discrepancies.
- Support User Acceptance Testing (UAT) and business validation.
Documentation
Prepare and maintain project documentation including:
- Source to target mapping
- Data lineage
- Technical specifications
- Data dictionaries
- Metadata documentation
- Validation and reconciliation reports
- Operational runbooks
Requirements
Experience
- Degree in Computer Science, Information Systems, Engineering or a related discipline.
- 3 to 5 years of relevant experience in data engineering, data migration, ETL development, reporting or data management.
- Experience within banking or financial services will be an advantage.