Senior Data Engineer - AI Infrastructure
Summary
Senior Data Engineer designing, building, and optimizing scalable data pipelines, warehouses, and lakes that power a client's AI/ML initiatives. Works closely with data scientists using Spark, Kafka, Airflow, SQL/NoSQL databases, cloud data services (AWS/Azure/GCP), and Python or Scala. On-site role in Queenstown.
About the Role
Our client, a leader in data-driven innovation, is seeking a highly experienced Senior Data Engineer to build and maintain the robust infrastructure powering their AI initiatives in Queenstown, Eastern Cape . This role is critical for ensuring the seamless flow of data required for machine learning model training and deployment. You will be responsible for designing, implementing, and optimizing scalable data pipelines, data warehouses, and data lakes. The ideal candidate will possess a strong background in data engineering principles, cloud platforms, and experience working closely with data scientists and AI engineers. This is an excellent opportunity to play a key role in scaling our AI capabilities and working with a talented team dedicated to leveraging data for business success. Join us and contribute to building the foundation for advanced AI solutions.
Key Responsibilities
- Design, build, and maintain scalable and reliable data pipelines for ingesting, transforming, and processing large volumes of data.
- Develop and manage data warehouses, data lakes, and other data storage solutions.
- Implement ETL/ELT processes to ensure data quality, consistency, and availability.
- Optimize data infrastructure for performance, cost-efficiency, and scalability to support AI/ML workloads.
- Collaborate with Data Scientists and AI Engineers to understand their data requirements and provide necessary data solutions.
- Implement data governance, security, and compliance best practices.
- Automate data infrastructure operations and monitoring.
- Troubleshoot and resolve data-related issues promptly.
- Stay current with emerging technologies in data engineering and cloud platforms.
Requirements
- Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field.
- 5+ years of professional experience in data engineering.
- Proven experience building and managing data pipelines using tools like Apache Spark, Kafka, Airflow.
- Strong proficiency in SQL and NoSQL databases.
- Hands-on experience with cloud platforms such as AWS, Azure, or GCP, including their data services.
- Experience with data warehousing concepts and technologies (e.g., Redshift, Snowflake, BigQuery).
- Familiarity with big data technologies and distributed systems.
- Understanding of AI/ML concepts and data requirements is highly desirable.
- Excellent programming skills in Python or Scala.
- Strong analytical and problem-solving skills.
Benefits
- Competitive salary and performance bonuses.
- Comprehensive health, dental, and vision insurance.
- Generous paid time off and holidays.
- Opportunities for professional development and advanced training.
- A collaborative and innovative work environment in Queenstown, Eastern Cape .
- Retirement savings plan with company match.
- Support for continuous learning and skill enhancement.