Senior Data Engineer
Summary
Build and maintain the core data pipelines and platform infrastructure for a global logistics company, ensuring scalable ingestion, governance, and performance of enterprise data systems.
Design and build the core data processing pipelines that power the Data Platform, enabling ingestion, integration, and transformation of data from source systems.
Develop and maintain the platform's data ingestion mechanisms, both streaming and batch/replication-based, bringing source data into the platform's raw, cleansed, and curated layers.
Implement and maintain data governance capabilities within the platform, including cataloging, access control policies, and data masking mechanisms, in line with platform standards.
Translate architectural decisions into concrete, production-ready platform components and configurations.
Optimize the performance, scalability, and cost-efficiency of the platform's data processing and storage infrastructure.
Design and maintain CI/CD pipelines for the automated testing, deployment, and versioning of platform code and configurations.
Monitor platform health and data pipeline performance through the platform's observability tooling, ensuring compliance with agreed service levels.
Diagnose and resolve platform-level incidents, pipeline failures, and performance issues, coordinating remediation with teams consuming the platform.
Support the onboarding of new data sources onto the platform, including technical validation, configuration, and adherence to platform standards.
Prepare and maintain technical documentation, operational runbooks, and best-practice guides for the platform.
Collaborate with the Data Governance function to ensure accurate metadata, lineage, and cataloging of platform data assets.
Monitor and help control infrastructure and compute costs related to the platform, proactively flagging anomalies.
Design and implement automated data quality checks and validation frameworks to safeguard the integrity of platform data.
Participate in code reviews and mentor other engineers on platform engineering standards and best practices.
Evaluate and pilot new platform technologies and capabilities, in coordination with the Data Architecture function.
Ensure security, privacy, and regulatory compliance across platform components, including access controls and data masking.
Participate in incident response and contribute to post-incident reviews for platform outages and cost anomalies.
Provide technical support and consultation to teams building solutions on top of the platform.
Automate routine platform maintenance tasks, including storage optimization and data lifecycle management.
Contribute to the definition and enforcement of platform engineering standards and naming conventions.
Participate in on-call and support rotations to ensure continuity of platform operations.
Perform other duties as assigned by the department manager related to the development and maintenance of the platform.