Staff AI Engineer
Posted Updated
Description
- Grafana Labs is seeking a Senior Engineer (AI & Automation) to own the AI agent infrastructure and automation platform that powers our Marketing Operations organization
- You’ll build multi-agent architectures, LLM integrations, and backend services that connect AI models to internal and third-party data platforms. You’ll ship production systems that teams depend on daily
- This is a high-autonomy role where you own the technical direction. You’ll identify the highest-leverage problems across Marketing, RevOps, and SDR teams, design the solutions, and ship them
- You’ll define the technical direction for the automation platform (data models, API contracts, shared libraries, reference architectures) and partner with Data Engineering, GTM Systems, and Field Operations to build scalable, self-service automation that eliminates manual work and drives operational efficiency
- Agentic Systems & AI Infrastructure
- Own end-to-end development of multi-agent AI systems, from architecture and implementation through testing, deployment, and ongoing operation
- Build modular, composable agentic systems using orchestration frameworks (LangChain, CrewAI, Anthropic MCP, or similar) that operate 24/7 across teams
- Develop reusable agentic skills that agents invoke across interfaces (Slack, dashboards, internal apps, CLIs)
- Implement observability and feedback loops including logging, performance metrics, prompt iteration, model evaluation, and cost management
- Establish governance and compliance standards for AI workflows including access controls, audit trails, PII handling, and human-in-the-loop escalation paths
- Systems Integration & Backend Services
- Build MCP servers, APIs, CLIs, and microservices connecting AI models to business systems (BigQuery, Slack, CRMs, email, calendars, analytics tools)
- Architect data flows for retrieval-augmented generation (RAG), connecting LLMs to internal knowledge bases, customer data, and real-time business context
- Build serverless or containerized services (GCP Cloud Functions, Cloud Run) that scale with usage and integrate with Grafana’s cloud infrastructure
- Automation & Workflow Enablement
- Partner with RevOps, Demand Generation, Regional Marketing, and SDR teams to scope high-impact automation problems, identify bottlenecks, and build solutions with measurable business outcomes
- Design and deploy workflows using orchestration tools (n8n, Workato, or custom platforms) with CI/CD, testing, and production reliability standards
- Build systems designed for self-service with documentation, playbooks, and enablement materials that let partner teams operate independently
- We invest heavily in developer productivity. You’ll have access to AI coding assistants (Claude Code, Gemini CLI, OpenAI Codex, and others of your choice within security guidelines). We encourage pragmatic AI-assisted development paired with strong code review and quality standards
Benefits
- Vacation: Balance is key. Our team enjoys 30 days of paid vacation each year on top of national holidays, parental leave, and sick leave. We also take a breather on a number of Grafana Shutdown Days each year
- Healthcare: We’re proud to provide health coverage or stipends for our colleagues in the US, UK, Canada, the Netherlands, Sweden, Singapore, and India
- Retirement planning: There’s no time like the present to start saving for your future. We make employer contributions into the pension pots of our team members in the US, UK, Canada, the Netherlands, Sweden, and Germany
- Professional development: On top of a $1,500 annual learning and development stipend, Grafanistas have thousands of on-demand courses at their fingertips to help them grow professionally. Want to attend a conference or training? Go ahead. Just pass on what you learned
- Work location: Vast majority of our roles are fully remote, focused on hiring the best talent and allowing you to perform from the comfort of your home. If you fancy a change of scene, we’ll also reimburse you up to $175 a month for a personal co-working space
- Choice of tech: There’s no one-size-fits-all when it comes to the tech required to do your job. Choose the laptop and accessories you need when you join us, and we’ll refresh them every three years
- Mindfulness: When you join the team, you can sign up for a complimentary subscription to Headspace to take advantage of the benefits of mindfulness and meditation. Our wellbeing resource group also organize sessions run by fellow Grafanistas or external trainers
- Global Employee Assistance Program: We offer all team members a 100% confidential support service with 24/7 365 access to professionally qualified counsellors and specialists
- Paid parental leave: Grafana offers paid parental leave to all eligible new parents. This offers Grafanistas time to bond with and care for their children in the first year after birth or adoption
- Experience building and operating multi-agent systems at scale including agent decomposition, orchestration patterns (sequential chains, router/dispatcher, parallel fan-out), state management, and production monitoring
- Fluent with AI-assisted development tools (GitHub Copilot, Cursor, Claude Code). You use AI to build AI systems
- Proven ability to identify high-leverage problems, push back on low-impact requests, and deliver end-to-end with minimal direction
- Hands-on experience with LLM frameworks and patterns including prompt engineering, RAG, function calling/tool use, structured output parsing, and evaluation
- Strong proficiency in Python and JavaScript/Node.js with Git-based workflows, code review practices, and testing discipline
- 8+ years of software engineering experience with depth in backend development, systems integration, or data/analytics engineering
- 2+ years hands-on experience applying LLMs/AI to production workflows, not just prototypes
- You diagnose business problems before writing code. You think in workflows and outcomes, not just functions
- Deep familiarity with Google Cloud Platform, BigQuery, and serverless/containerized services (Cloud Functions, Cloud Run)
- Understanding of LLM failure modes and production mitigations including confidence thresholds, fallback logic, human escalation, and cost/latency management
- Clear technical communicator who can explain complex systems in simple terms to both engineers and business stakeholders
- Experience with vector databases or retrieval pipelines (Pinecone, Weaviate, ChromaDB, Qdrant, pgvector)
- Familiarity with marketing or sales platforms (Salesforce, Customer.io, HubSpot, Marketo, Outreach)
- Experience with frontend frameworks (React, Slack Block Kit) for building user-facing AI tool interfaces
- Observability tooling for AI systems (LangSmith, Weights & Biases, custom evaluation frameworks)
- Experience with workflow orchestration platforms (n8n, Temporal, Prefect, Airflow)
- Familiarity with Model Context Protocol (MCP) or similar standards for connecting AI systems to data sources
- Prior work automating marketing, sales, or customer success workflows in a B2B SaaS environment
- Active in open-source communities. Grafana is built on OSS and we value engineers who share that DNA
Skills
- Agentic AI
- AI
- Airflow
- Analytics
- Anthropic
- API
- Automation
- BigQuery
- ChromaDB
- CI/CD
- Claude Code
- CLI
- Cloud
- Cloud Run
- CrewAI
- Customer.io
- Data Engineering
- Demand Generation
- GCP
- Git
- GitHub
- Github Copilot
- Grafana
- HubSpot
- JavaScript
- LangChain
- LangSmith
- LLM
- Marketo
- MCP
- Microservices
- Model Evaluation
- n8n
- Node.js
- Observability
- OpenAI
- pgvector
- Pinecone
- Prefect
- Prompt Engineering
- Python
- Qdrant
- RAG
- React
- SaaS
- Salesforce
- Serverless
- Slack
- System Integration
- Vector Databases
- Weaviate
- Workato
- Workflow Orchestration
Sign in to read and join the discussion.