Data Engineer - Big Data Platforms
Summary
A data engineer builds and maintains big data infrastructure in Upington: designing scalable data pipelines, data warehouses, and data lakes, ensuring data quality and performance. Core tech includes Spark/Hadoop/Flink, SQL and NoSQL databases, Python/Scala/Java, with cloud data services as a plus.
About the Role
Our client is seeking a dedicated Data Engineer to build and manage their big data infrastructure in Upington . You will be responsible for designing, constructing, and maintaining scalable data pipelines and platforms that enable efficient data processing, storage, and analysis. This role is crucial for ensuring the availability and integrity of data assets, supporting data scientists and analysts in their work. If you have a passion for working with large-scale data systems and driving data-driven insights, this is an excellent opportunity.
Key Responsibilities
- Design, build, and optimize robust and scalable data pipelines for ingesting, transforming, and processing large volumes of data.
- Develop and maintain data warehousing solutions and data lakes using modern big data technologies.
- Ensure data quality, integrity, and security across all data platforms.
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
- Monitor data infrastructure performance, troubleshoot issues, and implement optimizations.
- Stay current with emerging big data technologies and best practices.
Requirements
- Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
- 3+ years of experience as a Data Engineer or in a similar role involving big data technologies.
- Proficiency with distributed data processing frameworks such as Spark, Hadoop, or Flink.
- Experience with SQL and NoSQL databases, as well as data warehousing concepts.
- Knowledge of cloud platforms (AWS, Azure, GCP) and their data services is a plus.
- Strong programming skills in Python, Scala, or Java.
Benefits
- Competitive salary and performance-based bonuses.
- Comprehensive health and retirement benefits.
- Opportunities for training and professional development in data engineering.
- A collaborative environment focused on data innovation in Upington .
- Contribution to building and maintaining critical data infrastructure.