Senior Data Engineer - Big Data Solutions
Summary
A senior data engineer based in the Klerksdorp office will design, build, and maintain scalable ETL/ELT pipelines, data warehouses, and data lakes for a client, optimizing big data frameworks like Spark and Hadoop while ensuring data quality and governance to support analytics and machine learning.
About the Role
Our client is seeking a highly experienced Senior Data Engineer to lead the design and implementation of their big data infrastructure. Based in our Klerksdorp office, this role is crucial for building and maintaining robust data pipelines, ensuring data quality, and enabling advanced analytics and machine learning initiatives. You will work with large-scale datasets, optimize data processing frameworks, and collaborate with data scientists and analysts to deliver reliable data solutions. This is a fantastic opportunity for a seasoned data professional to shape the future of data at the organization.
Key Responsibilities
- Design, build, and maintain scalable and efficient ETL/ELT data pipelines using big data technologies.
- Develop and manage data warehousing solutions and data lakes to support business intelligence and analytics needs.
- Optimize data processing frameworks (e.g., Spark, Hadoop) for performance and cost-efficiency.
- Ensure data quality, integrity, and security across all data platforms.
- Collaborate with data scientists and analysts to understand data requirements and provide accessible, high-quality data.
- Implement data governance policies and procedures to maintain compliance and data integrity.
Requirements
- Minimum of 5 years of experience in data engineering or a related field.
- Proven experience with big data technologies such as Apache Spark, Hadoop, Kafka, or Flink.
- Strong proficiency in SQL and programming languages like Python or Scala.
- Experience with cloud-based data platforms (e.g., AWS Redshift, Snowflake, BigQuery).
- Deep understanding of data warehousing concepts and best practices.
- Excellent problem-solving skills and ability to work independently and in a team.
Benefits
- Competitive salary and comprehensive benefits package.
- Opportunity to work on challenging big data projects in Klerksdorp .
- Health, dental, and vision insurance.
- Generous paid time off and retirement savings plan.
- Continuous learning opportunities and professional development support.