Principal Core Infrastructure Engineer

You will lead the architecture and development of scalable, elastic distributed systems, driving performance, reliability, security, and operational excellence. You will define scalability requirements and SLOs, design fault-tolerant services that support upgrades without downtime, and establish telemetry and rigorous resilience testing. You will also resolve complex production issues, mentor engineers, and build infrastructure automation for safe deployments, patching, and rollbacks.

Key Responsibilities
System Design & Architecture - System Scalability:
–Lead the development and implementation, and begin to architect, components of scalable distributed systems that support horizontal and vertical scaling to meet system demands, including leveraging distributed state management tools.
–Optimize code and/or systems for large-scale data processing and high-throughput requirements to support hyper-scale systems.
–Define scalability requirements for owned components and ensure design and implementation requirements are met.
–Design systems to scale with elasticity (e.g., effectively scaling both up and down).
–Leverage data plane platforms to effectively handle large-scale data retrieval, storage, and processing.
–Design performance and load testing.
System Design & Architecture - System Reliability Design:
–Build and design fault-tolerant components and systems capable of withstanding in-service updates by implementing redundancy, replication, and automatic failover mechanisms.
–Design systems to effectively handle service disruptions (e.g., network partitions) by prioritizing consistency, availability, or partition tolerance.
–Implement and optimize approaches to handle network unreliability, including load-shedding, throttling, and rate-limiting.
–Design components and systems that are durable and adhere to service level objectives (SLOs), setting expectations for availability and durability of other computing services within the department.
System Design & Architecture - System Reliability Performance:
–Define key performance indicators (KPIs) and telemetry to identify gaps or issues in running systems.
–Build and customize moderately complex dashboards, telemetry systems, and alerting mechanisms to proactively monitor components and system health..
System Design & Architecture - Correctness / Availability:
–Design and implement functional and correctness requirements for feature sets and/or systems in new or existing systems.
–Design complex test scenarios (e.g., fault-injection, brown-out) to evaluate system correctness.
–Implement data replication and synchronization techniques to maintain data integrity and availability.
Operational Troubleshooting & Incident Management:
–Take a proactive role in diagnosing, debugging, and resolving issues in active components and systems to support ongoing operation, and mentor others in these processes.
–Implement strategies to prevent interruptions, ensuring no maintenance windows are required for customers and users when resolving issues.
–Maintain expertise in owned components and systems to ensure effective troubleshooting and performance.
–Meet operational readiness expectations through design and implementation.
–Serve in operational support rotations, providing guidance in incident response and root cause investigations.
Compliance & Security:
–Implement robust security measures to protect data and applications in multi-tenant environments, including encryption techniques and access controls.
–Execute remediation plans to address identified security gaps.
–Ensure cloud infrastructure is in compliance with industry standards and regulations and that documentation is up to date.
Automation & Change Management:
–Develop and maintain automation scripts and tools (e.g., Infrastructure as Code (IaC)) to manage cloud infrastructure.
–Create and adhere to change management plans for patching, updating, and rolling back applications, and begin designing systems and components to allow for automation of these processes.

Basic Qualification

7+ years of industry experience working on commercial or open-source software.

Strong fluency in C++ or Java or similar modern languages

Excellent problem-solving skills and a deep understanding of Computer Science fundamentals, including data structures, algorithms, and distributed systems.

Experience with cloud infrastructure (AWS/GCP/Azure) and build control plane systems for cloud scale

Strong systems programming skills, including experience with multi-threading and concurrency.

A solid foundation in distributed systems concepts (e.g., consistency, scaling, failure recovery).

Experience with implementation testing, debugging, and documentation, specifically within a Linux development environment.

BS in Computer Science; Masters preferred

Intense curiosity, willingness to question and passion for making systems

Experience designing architectures that demonstrate deep technical depth in one area, or span many products, to enable high availability, scalability, market-leading features and flexibility to meet future business demands

Preferred Qualifications

Experience as technical lead on a large scale cloud service
Hands-on experience developing and maintaining services on a public cloud platform (e.g., OCI, AWS, Azure)
Experience working on Kubernetes
Knowledge of Infrastructure as Code (IAC) languages, preferably Terraform
Strong knowledge of databases (SQL and NoSQL)
Strong knowledge of Computer Networking (OSI layers, HTTP, DNS, TCP/IP, DHCP, Routers, Gateways, Subnets, etc.)
Knowledge of Linux internals, Linux/Unix troubleshooting skills
Familiarity with host virtualization technologies (KVM, Containers, Docker, etc.)
Able to effectively communicate technical ideas verbally and in writing (technical proposals, design specs, architecture diagrams and presentations)
Experience with hiring, mentorship and raising the talent bar across the organization