Data Engineer
Posted Updated
Description
Eqplus Technologies (Pty) Ltd in Johannesburg leads data initiatives and seeks a senior data engineer to maintain the data warehouse, design ETL/ELT processes, and optimize machine learning environments.
You will implement solutions in AWS, develop automated pipelines, perform root cause analysis, and provide technical leadership across structured and unstructured data, working in SA's data landscape.
Responsible for solution design and development of various functionalities in AWS for the project flow. Develop and maintain automated ETL pipelines (with monitoring) using scripting languages such as Python, SQL, and AWS services. Develop Glue Jobs for batch data processing and create Glue Catalog for metadata synchronization. Develop data pipelines using AWS Lambda and Step Functions for data processing. Manage data coming from different sources. Experience with AWS services to manage applications in the cloud and create or modify instances. Implement solutions using the Scaled Agile Framework (SAFe). Be involved in the performance and optimization of existing algorithms in Hadoop using Spark Context. Create Hive Tables, load data, and write Hive queries. Degree in Information Technology. 6-7 years of experience in a similar role. Experience with Talend. Knowledge of SSIS, SSAS, and SSRS. Experience with Clover ETL. Strong SQL background. Experience with AWS. Familiarity with Jupyter Notebook. Experience in Data Warehousing. Experience in Cloud Warehousing.Skills
Sign in to read and join the discussion.