Senior Data Engineer AI Focus
Summary
Designs and scales AI-focused data architectures (feature stores, vector databases, ML pipelines) while building high-reliability ETL/ELT pipelines for structured/unstructured data, ensuring data quality for AI models using tools like Spark, Kafka, BigQuery, and Vertex AI.
Virtusa helps its Clients by becoming a true extension of their software and data development capabilities. Through the readily set up, comprehensive, and self-governing teams, we let our Clients focus on their business while we make sure that their software products and data tools scale up accordingly and with outstanding quality.
We are looking for experienced team players to fill the position of Senior Data Engineer (AI Focus) and participate in our up-and-coming project for our client from the high-tech area. This is a project for consultants working remotely from Poland.
You can expect:
- AI Data Infrastructure: Design and scale data architectures specifically tailored for Machine Learning (ML) lifecycles, including feature stores, vector databases, and model training pipelines.
- Pipeline Engineering: Architect, build, and maintain robust ETL/ELT pipelines that handle structured and unstructured data with a focus on low latency and high reliability.
- Data Governance & Quality: Implement automated testing and monitoring for data integrity, ensuring that "garbage in, garbage out" is never a reality for our AI models.
- Proficiency in Python and SQL. Experience with Java or Scala is a plus.
- Big Data Tools: Hands-on experience with frameworks like Apache Spark, Flink, or Kafka.
- AI/ML Familiarity: Experience supporting AI projects (e.g., handling embeddings, managing datasets for LLM fine-tuning, or working with tools like LangChain or LlamaIndex).
- Infrastructure as Code: Comfortable with Terraform or Docker/Kubernetes to manage data environments.
- BigQuery: Advanced optimization and partitioning strategies.
- Vertex AI: Experience building and deploying data pipelines within the Vertex ecosystem.
- Dataflow/Dataproc: Professional experience with managed Apache Beam or Spark services.
- Building real-time event-driven architectures.
- APIs: Understanding of REST APIs and microservices architecture.