AI Engineer

Full-time, Hybrid / Remote, Platform Engineering

ABOUT THE ROLE

You will build and operate a self-hosted AI platform that runs entirely inside customer-controlled infrastructure — no dependency on external model APIs. The platform serves open-weight models, exposes a governed gateway for applications and agents, and enforces privacy and safety controls in-line with every request. Customers adopt it because their data cannot leave their jurisdiction or their network, so correctness, isolation, and auditability matter as much as model quality.

This is an applied engineering role, not a research role. You will not be training foundation models. You will be serving them efficiently, wrapping them in retrieval and tooling, measuring whether they actually work, and making the whole thing run reliably on a fixed GPU budget.

KEY RESPONSIBILITIES

• Deploy and tune open-weight LLMs for production inference — model selection, quantisation, batching, KV cache and context configuration, and throughput/latency optimisation on constrained GPU capacity.
• Build and maintain the model gateway layer: routing, fallback, rate limiting, per-tenant quotas, cost and token accounting, and API compatibility for downstream applications.
• Design and ship retrieval pipelines end to end — ingestion, chunking, embedding, vector and hybrid search, reranking, and evaluation of retrieval quality against real customer corpora.
• Build agent workflows with durable execution: multi-step tool use, human-in-the-loop checkpoints, retries, and recovery from partial failure.
• Implement the guardrail control plane — PII detection and redaction, prompt injection and jailbreak defence, content classification, and policy enforcement on both input and output paths.
• Integrate tool access through a governed MCP gateway with per-tenant authorisation, so agents can only reach the tools and data a given tenant is entitled to.
• Build evaluation harnesses and regression suites for prompts, retrieval, and agent behaviour; instrument tracing and quality metrics so model or prompt changes are measured rather than guessed at.
• Package the stack for deployment across delivery modes — shared multi-tenant, dedicated single-tenant, and fully air-gapped on-premise appliances — with reproducible configuration.
• Work with the platform and security teams to meet data residency, retention, and audit requirements, and to keep evidence of controls ready for customer and regulator review.

REQUIRED QUALIFICATIONS

• 4+ years of software engineering experience, with at least 1–2 years shipping LLM-backed systems to production users.
• Strong Python, plus comfort in a typed backend language for service work (TypeScript, Go, or similar).
• Hands-on experience self-hosting model inference — vLLM, SGLang, TGI, or equivalent — including GPU memory management and performance tuning. Calling a hosted API is not a substitute.
• Practical RAG experience beyond a demo: you have measured retrieval quality, diagnosed why answers were wrong, and fixed it at the pipeline level.
• Experience building agent or multi-step LLM workflows with real error handling, not linear prompt chains.
• Solid grounding in containers, Kubernetes, and CI/CD; able to own your services in production.
• Working knowledge of PostgreSQL and vector search (pgvector, Qdrant, or similar).
• Clear written communication — architecture decisions, trade-offs, and incident write-ups.

NICE TO HAVE

• Experience with air-gapped, on-premise, or otherwise network-isolated deployments.
• Familiarity with LLM observability and evaluation tooling (Langfuse, Ragas, or equivalent), and with durable workflow engines (Temporal) and agent frameworks (LangGraph).
• Fine-tuning experience — LoRA/QLoRA, preference tuning, or domain adaptation of open-weight models.
• Exposure to identity and authorisation infrastructure (OIDC, Keycloak, policy engines such as Cerbos or OPA).
• Understanding of data protection and AI governance regimes — the Philippine Data Privacy Act and NPC issuances, sectoral regulator expectations, the EU AI Act, or NIST AI RMF.
• CUDA, Triton, or inference-kernel-level performance work.

WHAT SUCCESS LOOKS LIKE

• Inference capacity is measured, predictable, and cost-efficient — you know the tokens per second, the per-tenant cost, and where the ceiling is.
• Model, prompt, and retrieval changes ship behind evaluations, with regressions caught before customers see them.
• Guardrails are enforced consistently across every entry point and are demonstrable to an auditor.
• A new deployment — including an isolated on-premise one — can be stood up from configuration rather than tribal knowledge.

WORKING ARRANGEMENT

Full-time. Hybrid or remote with regular overlap for design and incident work. Occasional on-site time for customer deployments.