Data Engineer - Big Data Platforms
Summary
Build and maintain scalable big data platforms and ETL/ELT pipelines for a data-solutions technology firm in Paarl, Western Cape, enabling analytics and ML. Hands-on work with Spark/Hadoop/Kafka, cloud data services (AWS/Azure/GCP), SQL/NoSQL, and Python/Scala/Java in a hybrid role.
Our client, a rapidly growing technology firm specializing in data solutions, is seeking an experienced Data Engineer to join their team. This position is based in Paarl, Western Cape , and involves the design, development, and maintenance of robust, scalable big data platforms. You will play a key role in building and optimizing data pipelines that ingest, transform, and store vast amounts of data, enabling advanced analytics and machine learning initiatives across the organization. The ideal candidate has a strong background in data architecture, distributed systems, and programming, with hands-on experience with cloud-based data technologies. This is an excellent opportunity to work with cutting-edge technologies in a dynamic and collaborative environment, contributing directly to the company's data infrastructure and strategy. You will be instrumental in ensuring data availability and quality for critical business decisions.
Key Responsibilities- Design, build, and maintain scalable and reliable big data pipelines using distributed computing frameworks.
- Develop and optimize data ingestion, transformation, and loading processes.
- Implement data warehousing solutions and manage data storage systems.
- Ensure data quality, integrity, and security across all data platforms.
- Collaborate with data scientists, analysts, and other engineers to understand data requirements and deliver solutions.
- Monitor and troubleshoot data pipeline performance, identifying and resolving issues proactively.
- Implement and manage ETL/ELT processes.
- Stay current with emerging big data technologies and best practices.
- Document data architectures, processes, and standards.
- Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
- Proven experience as a Data Engineer or in a similar role, with a focus on big data technologies.
- Strong proficiency in SQL and experience with various database technologies (relational, No SQL).
- Hands-on experience with big data technologies such as Apache Spark, Hadoop, Kafka, or similar.
- Experience with cloud platforms (AWS, Azure, or GCP) and their data services (e.g., S3, Redshift, Data Factory, Big Query).
- Proficiency in programming languages like Python, Scala, or Java.
- Understanding of data modeling, data warehousing concepts, and ETL/ELT principles.
- Experience with workflow management tools (e.g., Airflow) is a plus.
- Excellent problem-solving and analytical skills.
- Strong communication and collaboration abilities.
Our client offers a competitive salary package, complemented by a comprehensive benefits scheme. This includes medical aid contributions, a provident fund, and generous paid vacation time. We encourage professional development through training programs, workshops, and opportunities to attend industry conferences. The role is situated in the beautiful Paarl, Western Cape region, offering a stimulating work environment. The hybrid nature of this position allows for flexibility between working from home and collaborating in the office. Additional benefits include performance bonuses and a supportive team culture focused on innovation and continuous improvement.
Skills
Sign in to read and join the discussion.