Lead Data Engineer - Big Data Solutions

About the Role

Our client, a leader in data analytics and business intelligence, is seeking a visionary Lead Data Engineer to spearhead their Big Data Solutions initiatives. This pivotal role is based in Upington, Northern Cape , and involves designing, building, and maintaining robust data pipelines and infrastructure to support advanced analytics and machine learning capabilities. You will lead a team of data engineers, define best practices, and ensure the efficient and reliable flow of data across the organization. The position offers a hybrid work arrangement, combining on-site collaboration with remote flexibility. We are looking for an experienced leader passionate about leveraging data to drive business insights and innovation.

Key Responsibilities

• Lead the design, development, and implementation of scalable big data solutions and data warehousing systems.
• Build and maintain efficient, reliable, and robust ETL/ELT data pipelines using various big data technologies.
• Oversee the architecture and implementation of data lakes and data warehouses.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions.
• Mentor and guide a team of data engineers, fostering technical excellence and professional development.
• Ensure data quality, integrity, and security across all data platforms.
• Optimize data processing performance and data retrieval efficiency.
• Evaluate and adopt new big data technologies and tools.
• Develop and enforce data governance policies and best practices.
• Troubleshoot and resolve complex data pipeline and infrastructure issues.

Requirements

• Master's or Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
• 7+ years of experience in data engineering, with at least 2 years in a lead or senior capacity.
• Extensive experience with big data technologies such as Hadoop, Spark, Kafka, and Flink.
• Proficiency in programming languages like Python, Scala, or Java for data processing.
• Strong SQL skills and experience with relational and NoSQL databases.
• Experience with cloud platforms (AWS, Azure, GCP) and their big data services (e.g., S3, Redshift, BigQuery, Databricks).
• Familiarity with data warehousing concepts and dimensional modeling.
• Experience with data pipeline orchestration tools (e.g., Airflow, Luigi).
• Excellent leadership, communication, and problem-solving skills.
• Experience working in the Upington, Northern Cape region or willingness to work a hybrid schedule is preferred.

Benefits

Our client offers a highly competitive salary, performance-based bonuses, a comprehensive health benefits package, and significant opportunities for professional growth and skill development in the rapidly evolving field of big data. The hybrid work model supports work-life balance. Employees will have the chance to work with cutting-edge technologies, lead impactful data initiatives, and contribute to data-driven decision-making within a forward-thinking organization.