Data Engineer
Posted
Description
- Data Ingestion & Pipeline Engineering: Design, develop, test, and deploy automated ETL/data pipelines in Informatica and Snowflake to ingest data from partner state agencies and third parties into the SLDS common data model.
- Transformation & Analytics Layer Support: Implement transformation logic to convert raw agency feeds into harmonized, linked longitudinal records and analytics layer data models, adhering to core naming and modeling standards established with the Data Architect.
- Record Linkage & Identity Matching: Execute the SLDS internal matching logic to link newly ingested agency records to master longitudinal student and workforce profiles, handling match exceptions and cross-agency reconciliations.
- Production Pipeline Operations & Incident Response: Maintain operational ownership of scheduled feed runs, monitoring job execution, resolving run failures, reloading late or corrected files, reconciling row counts, and rebuilding downstream objects.
- Data Quality & Governance Compliance: Implement L1 (structural/format), L2 (completeness/de-duplication), and L3 (referential integrity/logic) validation checks, layout-change detection, and audit trails; enforce data privacy, suppression rules, and compliance standards (FERPA, COPPA, ADA, and state statutes).
- Performance Optimization & Platform Stewardship: Optimize query execution, warehouse compute, and pipeline performance for cost and latency; identify and tune heavily used models and performance hot spots in Snowflake.
- Quality Assurance, Code Review & Promotion: Adhere to the unified release process across Development, Test, UAT, and Production environments using Azure DevOps and Azure Test Plans; author and execute test cases, conduct peer code reviews, and maintain release change logs.
Skills
Sign in to read and join the discussion.