Senior Data Engineer

Position Overview

Index Analytics is seeking a Sr. Data Engineer to support Government clients to design, build, and optimize scalable cloud-based solutions, data pipelines, and implement connections to knowledge sources. The Sr. Data Engineer plays a key role in modernizing the organization’s data ecosystem by helping transition legacy solutions to a contemporary infrastructure.

As part of a cross-functional team including Data Engineers, health policy researchers, Analysts, the engineer will support efforts to design and implement a robust environment capable of ingesting diverse data sources to support advanced analytics and reporting needs. Core responsibilities include defining structural, interface, and business requirements for data solutions; designing relational and non-relational databases and their associated integration components; and implementing Python based automated data pipelines.

This role blends advanced data engineering with hands‑on cloud solutions engineering, leveraging AWS, Databricks and modern DevOps practices. The ideal candidate has experience delivering high‑quality solutions in an Agile environment.

Responsibilities

• Collaborate closely with stakeholders, cross‑functional and internal technical teams to gather requirements, document business rules and develop a thorough understanding of the business context and objectives.
• Configure and manage connections from analytic tools to back end data sources, repositories, and platforms including Databricks and various APIs.
• Oversee Databricks unity catalog population and administration.
• Formulate and document technical and coding standards.
• Collaborate to design secure, scalable, and cost‑optimized data solutions.
• Design, build, and maintain scalable, reliable ETL/ELT data pipelines using AWS and Databricks.
• Develop and optimize data models, both conceptual and physical, to support analytics, reporting, and operational consumption.
• Implement data quality, validation, and monitoring frameworks to ensure accuracy and reliability.
• Ensure data workflows are modular, testable, and properly version‑controlled.
• Operationalize pipelines with monitoring, alerting, and automated recovery mechanisms.
• Conduct advanced data analysis using languages such as Python and SQL.
• Develop documentation to include data models, data dictionaries, and data usage guides.
• Improve end-to-end performance of data workflows.
• Build and maintain CI/CD pipelines using GitHub to support automated testing, deployments, and continuous integration.
• Meet schedule deadlines and commitments with a high-level of quality of deliverables.
• Collaborate with a team of cross-functional resources in an Agile delivery environment to deliver iterative value.