Walk-in || L3 Observability

Virtual Walk-In Interview - IT Engineers

Saturday, October 31 • 10:00am 4:30pm

Google Meet joining info

Video call link: https://meet.google.com/fae-phgb-nyy

Key Responsibilities

• Serve as the final escalation and technical authority for complex, cross-domain incidents

involving application, infrastructure, network, and third-party dependencies.

• Own the end-to-end observability architecture and strategy in Datadog, including

instrumentation standards, tagging strategy, dashboard governance, and monitor design

principles.

• Lead root cause analysis for major/critical incidents and drive post-incident reviews,

ensuring permanent fixes and preventive monitoring are implemented.

• Define and continuously improve monitoring coverage for new and existing services,

including 3-tier applications, microservices, and cloud-native workloads.

• Design and govern SLOs/SLIs aligned to business-critical services, and advise application

and platform teams on instrumentation best practices.

• Drive automation initiatives: Datadog-as-code (Terraform/API-based configuration

management), auto-remediation workflows, and integration with CI/CD pipelines.

• Partner with application, infrastructure, network, and security teams to close monitoring

gaps and reduce mean time to detect (MTTD) and mean time to resolve (MTTR).

• Own capacity planning and performance engineering insights derived from long-term

observability data trends.

• Manage Datadog platform health: cost optimization, license/usage governance, data

retention, and integration management.

• Mentor L1 and L2 engineers, review their RCA quality, and build enablement material

(runbooks, training, best practices).

• Act as a key stakeholder in client governance calls, presenting trends, risks, and

improvement roadmaps.

• Evaluate and pilot new Datadog features or complementary observability tools, and lead

their rollout.

• Define escalation matrices and coordinate cross-team response during major outages.

Required Skills and Experience

• 6+ years of experience in Managed Services, SRE, or platform engineering roles, with at

least 3 years of hands-on ownership of a Datadog (or equivalent) observability platform

at scale.

• Expert-level knowledge of Datadog, including:

• Platform architecture: organization/team structuring, RBAC, tagging strategy, and multi-

environment setups.

• APM: distributed tracing at scale, service dependency mapping, and instrumentation

strategy across polyglot environments.

• Synthetics and RUM: designing comprehensive digital experience monitoring strategies

tied to business KPIs.

• Log Management: pipeline architecture, log-based metrics, and cost/volume governance.
• SLOs, error budgets, and anomaly/forecast-based alerting at an advanced level.
• Deep understanding of infrastructure and cloud architecture: compute, containers,

orchestration (Kubernetes), auto-scaling, and multi-cloud/hybrid environments.

• Deep understanding of networking: routing, load balancing, CDN, DNS, and complex

connectivity troubleshooting across hybrid environments.

• Strong incident and problem management leadership experience, including major

incident command and post-incident review facilitation.

• Experience defining and governing observability standards across multiple teams or

business units.

Application & Architecture Understanding (technology-agnostic)

• Expert-level understanding of 3-tier and N-tier application architectures, including how

failures cascade across presentation, application, and data layers.

• Expert-level understanding of cloud-native and distributed architectures: microservices,

event-driven systems, service meshes, API gateways, and asynchronous messaging.

• Ability to advise application teams on observability best practices (instrumentation,

tracing context propagation, structured logging) without needing to write code in a

specific language.

• Strong grasp of resiliency and failure patterns (retries, circuit breakers, backpressure,

caching, graceful degradation) and how to detect their failure modes through

observability data.

Good to Have

• Datadog advanced/expert certifications, or equivalent recognized observability

certifications.

• Strong scripting/automation background (Python, Terraform, or similar) for Datadog-as-

code and custom tooling.

• Experience with FinOps or cost governance for observability tooling at scale.
• Exposure to multiple observability ecosystems (New Relic, Dynatrace, Splunk,

Grafana/Prometheus, OpenTelemetry).

• Experience presenting to senior stakeholders or clients on observability maturity and

roadmaps.