Senior data engineer - big data solutions
Summary
A senior data engineer designs, builds, and maintains large-scale ETL/ELT pipelines, data warehouses, and data lakes using big data technologies like Spark, Hadoop, and Kafka plus cloud platforms such as AWS Redshift, Snowflake, or BigQuery, ensuring data quality and supporting analytics and ML from the Klerksdorp office.
Our client is seeking a highly experienced Senior Data Engineer to lead the design and implementation of their big data infrastructure. Based in our Klerksdorp office, this role is crucial for building and maintaining robust data pipelines, ensuring data quality, and enabling advanced analytics and machine learning initiatives. You will work with large-scale datasets, optimize data processing frameworks, and collaborate with data scientists and analysts to deliver reliable data solutions. This is a fantastic opportunity for a seasoned data professional to shape the future of data at the organization.
Key Responsibilities Design, build, and maintain scalable and efficient ETL/ELT data pipelines using big data technologies. Develop and manage data warehousing solutions and data lakes to support business intelligence and analytics needs. Optimize data processing frameworks (e.g., Spark, Hadoop) for performance and cost-efficiency. Ensure data quality, integrity, and security across all data platforms. Collaborate with data scientists and analysts to understand data requirements and provide accessible, high-quality data. Implement data governance policies and procedures to maintain compliance and data integrity. Requirements Minimum of 5 years of experience in data engineering or a related field. Proven experience with big data technologies such as Apache Spark, Hadoop, Kafka, or Flink. Strong proficiency in SQL and programming languages like Python or Scala. Experience with cloud-based data platforms (e.g., AWS Redshift, Snowflake, Big Query). Deep understanding of data warehousing concepts and best practices. Excellent problem-solving skills and ability to work independently and in a team. Benefits Competitive salary and comprehensive benefits package. Opportunity to work on challenging big data projects in Klerksdorp. Health, dental, and vision insurance. Generous paid time off and retirement savings plan. Continuous learning opportunities and professional development support.