Data Engineer (Secure Supply Chain/Anti-Counterfeit Platform)
Posted 6
views
Summary
Data engineer for an anti-counterfeit supply-chain platform: designs secure AWS data architecture spanning event streaming, S3 data lake, Snowflake warehouse, and transactional SQL databases, with Hyperledger provenance anchoring. Day to day: building scalable Python/SQL pipelines, encryption and access controls, curated datasets, and BI reporting.
Key Responsibilities:
Design and implement ascalable data architecturecombining:
- Event and data streaming
- Data lake (object storage, e.g. S3)
- Data warehouse (e.g. Snowflake)
- Transactional database (SQL or similar)
Definedatabase architecture for item identity, provenance and verification events
- Encryption at rest and in transit
- Field-level encryption for sensitive data
- Integration with KMS / HSM systems for key management
Integrate withpermissioned ledger systems(e.g. Hyperledger Fabric) for provenance anchoring
Enable analytics use cases by:
- Structuring data for efficient querying and reporting
- Building curated datasets and feature stores
SupportBI tools and dashboards(e.g. Power BI, Looker, Tableau)
Implementdata access layers (APIs, views)with role-based controls
Define and enforce:
- Monitoring and alerting for pipeline failures
Maintain data catalogue and documentation
Person Specification
- 5+ years indata engineering or backend data systems in AWS cloud environment
- Proven experience buildingdata pipelines at scale/streaming systems
- Strong SQL and experience withdata modelling (OLTP + OLAP)
- Hands-on experience with:
- Distributed data processing
- Data lakes and warehouses
Essential Technical Skills
- Strong programming skills (Python or equivalent) for data pipelines
- Experience withAPI-driven data systems
- Knowledge ofdata formats(Parquet, Avro, JSON)
- Understanding ofdata partitioning, indexing, and performance optimisation
- Understanding of:
- Encryption (at rest / in transit)
- Identity and access management (IAM)
- Secure handling of sensitive data
- Awareness ofdata integrity concepts:
- Hashing
- Digital signatures
- Audit trails
- Experience withblockchain / distributed ledger technologies(e.g. Hyperledger)
- Exposure tosupply chain systems or provenance tracking
- Experience withIoT or hardware-generated data
- Knowledge ofprivacy-preserving analytics(e.g. differential privacy, federated learning)
- Experience withregulated environments
Soft Skills
- Strong analytical thinking and problem-solving
- Ability to work across disciplines (software, hardware, security)
- Clear communication of complex technical concepts
- Attention to detail, especially around data correctness and integrity
Education
- Degree in Computer Science, Engineering, Mathematics, or related field