Associate Distinguished Engineer - Solution Architect
Summary
Designs and maintains a scalable data platform using Databricks/Unity Catalog, setting standards for ingestion, storage, and AI-driven consumption while aligning roadmaps with engineering and product teams.
Key Responsibilities
- Define and maintain the overall technical architecture for the KM data platform, spanning ingestion, storage (Databricks/Unity Catalog), transformation, security, and consumption layers.
- Translate multi-year program roadmaps into phased architectural plans, balancing near-term pilot needs against long-term scalability.
- Set architectural standards and guardrails for data modeling, pipeline design, and catalog registration, in partnership with the Senior Data Modeler and Data Engineers.
- Evaluate and recommend platform components, tools, and integration patterns (e.g., Unity Catalog vs. alternative catalogs, orchestration tools, vector search/embedding infrastructure).
- Lead technical design reviews and ensure alignment across data engineering, security/privacy, and downstream product teams (Knowledge Products, Research Products).
- Own non-functional requirements: scalability, performance, security, cost, and reliability of the platform architecture.
- Serve as the primary technical point of contact for architecture workshops and planning sessions (e.g., multi-day cross-functional architecture planning events).
- Assess technical risk and dependencies across workstreams and flag issues to the Engineering Manager and program leadership.
- Stay current on Databricks platform capabilities and enterprise AI/knowledge management architecture trends, bringing recommendations back to the team.
Required Qualifications
- 8+ years of experience in data/solution architecture roles, with significant hands-on or architectural experience in Databricks/Lakehouse environments.
- Deep understanding of Unity Catalog governance, data product design, and security classification models.
- Demonstrated experience architecting platforms that handle both structured and unstructured content at scale.
- Experience architecting for AI/LLM-driven consumption patterns (retrieval-augmented generation, vector search, agent-based data access) is highly valued.
- Strong track record of translating business/program roadmaps into technical architecture and staged delivery plans.
- Excellent stakeholder management skills — able to work across engineering, product, legal/privacy, and executive audiences.
- Experience leading architecture reviews and setting technical standards for a growing engineering team.
Preferred Qualifications
- Experience in professional services, consulting, or other knowledge-intensive industries.
- Familiarity with enterprise search/knowledge platforms (Glean, SharePoint, ServiceNow) and their integration patterns.
- Experience navigating legal/risk/privacy review processes for data platforms.
- Prior experience standing up a data platform team from a nascent or pilot stage to a scaled production capability. Success Metrics (First 6–12 Months)
- Documented target-state architecture for the KM data platform, validated with key stakeholders (Engineering Manager, Data Modeler, Product Owners).
- Architectural standards adopted across data modeling and engineering workstreams.
- Clear, staged technical roadmap aligned to the broader program timeline (e.g., through 2027).
Must have skills: Azure Data Factory, Data Modeling (Strong), Databricks