Ai Data Engineer - Large-scale Data Pipelines
Summary
Builds and maintains large-scale, scalable data pipelines and ETL/ELT systems that feed AI/ML models at a remote-first AI technology firm. Core stack spans big data tools (Spark, Kafka, Hadoop), cloud data platforms (AWS/Azure/GCP, Redshift, BigQuery), SQL, and Python/Scala; requires 5+ years of data engineering experience.
Our client, a cutting-edge technology firm specializing in AI-driven solutions, is looking for a skilled AI Data Engineer to join their fully remote team. This role is critical for designing, building, and maintaining robust, scalable data pipelines that power our advanced machine learning models and AI applications. You will work with vast datasets, ensuring data quality, availability, and accessibility for our data science and engineering teams. The ideal candidate will have a strong background in data engineering principles, big data technologies, and a solid understanding of the AI/ML lifecycle. This is an exceptional opportunity to contribute to impactful projects in a collaborative, remote-first environment.
Key Responsibilities- Design, construct, install, test, and maintain highly scalable data management systems and data pipelines for AI/ML workloads.
- Develop ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
- Optimize data structures and data flow for performance, efficiency, and cost-effectiveness.
- Collaborate with data scientists, ML engineers, and software engineers to understand data requirements and deliver data solutions.
- Implement data quality checks and monitoring mechanisms to ensure data accuracy and reliability.
- Work with distributed computing frameworks such as Apache Spark, Hadoop, and related technologies.
- Utilize cloud platforms (AWS, Azure, GCP) and their data services (e.g., S3, ADLS, GCS, Redshift, Big Query).
- Implement data governance and security best practices.
- Troubleshoot and resolve issues related to data pipelines and infrastructure.
- Stay current with emerging trends and technologies in data engineering and big data.
- Bachelor's degree in Computer Science, Engineering, Information Technology, or a related quantitative field.
- Minimum of 5 years of experience in data engineering, with a focus on building and managing large-scale data pipelines.
- Proven experience with big data technologies (e.g., Spark, Kafka, Hadoop Ecosystem).
- Strong proficiency in SQL and programming languages such as Python or Scala.
- Experience with cloud data platforms (AWS, Azure, or GCP) and their associated data services.
- Solid understanding of data warehousing concepts, data modeling, and database technologies (relational and No SQL).
- Familiarity with MLOps principles and tools is a plus.
- Experience with containerization (Docker) and orchestration (Kubernetes) is beneficial.
- Excellent problem-solving, analytical, and debugging skills.
- Strong communication and collaboration skills, especially in a remote setting.
- Ability to work independently and manage multiple priorities effectively.
Our client offers a competitive salary, comprehensive health and wellness benefits, and a generous retirement savings plan. As a remote-first company, employees enjoy the flexibility to work from their preferred location. Professional development is supported through ongoing training, access to online learning platforms, and opportunities to attend virtual conferences. The company culture in East London emphasizes innovation, collaboration, and employee well-being, fostering a productive and engaging work environment.
Skills
Sign in to read and join the discussion.