Senior Data Specialist / Data Engineer Freelancer
Summary
Build and maintain scalable data pipelines and quality frameworks using Spark, Databricks, Azure Data Factory, and streaming systems for chemical and scientific datasets.
Responsibilities
- Design and maintain scalable data pipelines (ingestion → transformation → publishing)
- Implement robust data quality frameworks and controls
- Support operational troubleshooting across large-scale datasets
- Work with streaming and batch processing systems
- Build and maintain data observability and monitoring solutions
- Support datasets at scale
Qualifications
- Strong experience with Spark / Databricks
- Hands-on expertise in Azure Data Factory
- Knowledge of streaming systems (Event Hubs / Kafka-compatible)
- Strong SQL and Python skills
- Experience with data quality frameworks and observability
- Background in large-scale data engineering environments
- Subject Matter Expert in: Chemical Data Pipelines at scale, heterogeneous scientific data, data quality & observability