Data Engineer
Description
About the role
The Data Engineer is responsible for managing, enhancing and progressively modernising Pentas Flora's enterprise data integration and data warehouse environment. You will develop and maintain pipelines that extract operational data from Core System and other business systems, transform and validate it using Python and ETL technologies, and load it into Microsoft SQL Server for reporting and analytics.
Key responsibilities
Develop and maintain Python-based ETL pipelines to extract, transform, validate and load data from Core System and other business systems.
Implement full and incremental data-loading processes with appropriate error handling, logging, reconciliation and recovery mechanisms.
Monitor scheduled pipelines, troubleshoot issues and ensure reliable data processing.
Design and maintain a Microsoft SQL Server-based data warehouse using dimensional modelling, fact and dimension tables, star schemas and historical data management.
Support layered architecture based on Medallion Architecture, or equivalent Raw, Staging, Curated and Business-ready layers.
Develop and optimise T-SQL queries, views and stored procedures to support efficient data processing and reporting.
Implement data quality controls covering completeness, accuracy, duplicates and source-to-target reconciliation.
Maintain data dictionaries, data mappings, business rules, data lineage and technical documentation.
Provide reliable and business-ready datasets for Microsoft Power BI and enterprise reporting.
Assess the existing Python ETL environment and identify opportunities to improve reliability, scalability and maintainability.
About you
Bachelor's Degree in Computer Science, Information Technology, Data Engineering or a relevant discipline.
Minimum 3 years of hands-on experience in data engineering, ETL or data warehousing.
Strong practical experience with Microsoft SQL Server, T-SQL and Python-based data processing.
Strong Python capability for production data extraction, transformation and loading.
Good understanding of dimensional modelling, ETL pipelines, incremental loading and data quality controls.
Experience designing or maintaining production ETL pipelines.
Practical experience with incremental loading, data validation, exception handling and reconciliation.
Understanding of layered data architecture such as Medallion Architecture or an equivalent approach.
Ability to troubleshoot data discrepancies and translate business reporting requirements into data solutions.
Experience with Microsoft Power BI, Microsoft Fabric, Azure Data Factory, SSIS, REST API integration or cloud data platforms is an added advantage.