Data Engineer 2
Summary
Designs and maintains data pipelines for enterprise systems, focusing on ingestion, transformation, and reliability using layered data models and CI/CD practices.
At HDR, our employee-owners are fully engaged in creating a welcoming environment where each of us is valued and respected, a place where everyone is empowered to bring their authentic selves and novel ideas to work every day. As we foster a culture of inclusion throughout our company and within our communities, we constantly ask ourselves\: What is our impact on the world?
Watch Our Story\:' https\://www.hdrinc.com/our-story'
Each and every role throughout our organization makes a difference in our ability to change the world for the better. Read further to learn how you could help make great things possible not only in your community, but around the world.
- Build and maintain batch and streaming ingestion pipelines from source systems into the enterprise data platform.
- Implement robust error handling, retries, and recovery patterns to improve reliability.
- Collaborate with platform teams on connectivity, access, and operational readiness.
- Develop transformation models using a layered structure (raw, conformed, curated) to separate concerns and improve maintainability.
- Use dependency management patterns to ensure correct build order and reproducible transformations.
- Leverage shared code and reusable components to standardize patterns, reduce duplication, and enforce consistent logic.
- Implement incremental processing where appropriate to reduce runtime and warehouse cost.
- Implement data quality tests (e.g., unique, not_null, accepted_values) on critical models and key business entities.
- Configure and monitor source freshness expectations for important upstream datasets.
- Maintain model and column-level documentation; publish documentation and lineage to enable impact analysis and change management.
- Integrate transformations into CI/CD workflows (build, test, docs generation) before promotion to higher environments.
- Participate in code reviews, follow branching standards, and adhere to quality gates and approval processes.
- Support release and rollback practices to maintain stability and reduce risk.
- Tune transformations for performance and efficiency; partner with stakeholders to manage cost impacts.
- Support monitoring and alerting for pipelines and transformations; respond to incidents and perform root cause analysis.
- Document operational procedures and contribute to runbooks/playbooks.