Data Engineer
Summary
Design and build scalable data pipelines and warehouses, ensuring reliable data flow and regulatory compliance using Python, PySpark, Kafka, and cloud tools.
- Relevant 3-year tertiary qualification BSC In Computer Science / Information Systems
In depth knowledge and understanding:
- Data governance and data management frameworks
- BI and Warehouse design
- Movement of data into the Cloud using the following tools Apache Kafka, Storm, Flume for ingesting data or Amazon Web Services (AWS) Cloud Development Kit (CDK)
- Operating with real-time streams, data warehouse queries, JSON, CSV, raw data
- Scripting Data Pipelines for scheduled movement of data
- Designing and developing ETL/ELT processes and data pipelines Experience working with Azure DevOps
- Exploratory Data Analysis - EDA
- SQL
- Microsoft SQL Server
- NoSQL
- Microsoft SQL Server
- Python (PySpark)
- C#, Json- calling APIs
- PowerShell
- Kafka
- Scala
- Splunk
- Elk Stack
- Data warehousing solutions
- Data Cleansing
- Terraform
Responsibilities
- Work as part of an agile data engineering team
- Ensure that data systems meet the companys regulatory commitments, and that data is reliable and efficient
- Support best business capabilities for high performance database solutioning
- Actively participate in team, cross-discipline and vendor-driven collaboration sessions or forums to increase understanding of the working environment by contribution and participation in the relevant Data Governance Frameworks
- Partner with Technology, Business and other stakeholders to ensure that database structures, programmes and applications meet business delivery requirements
- Monitor adherence to processes which support the prescribed data architectural frameworks and ensure development/delivery teams align to the required standards and methodologies
- Design and implement scalable end-to-end database solutions including:
- Addressing issues of data migration i.e. validation, clean-up and mapping and consistently apply data dictionaries
- Component design and development
- Identification and resolution of production and application development constraints
- Integration of new technologies and software into the existing landscape
- Development data set processes for data modelling, mining and production
- Optimal utilization of big data
- Documentation management and disaster recovery