Data Engineer
Summary
Data Engineer building scalable data platforms and pipelines for analytics and operational use cases, centered on Databricks, Delta Lake, Spark/pySpark, and cloud services on AWS and Azure, with Terraform-based IaC, CI/CD, and data quality/lineage practices. Day to day involves pipeline development, data modeling, optimization, and collaboration with scientists and analysts.
• Design, develop, and maintain scalable and reliable data platforms and pipelines for analytics and operational use cases
• Build distributed data processing workflows using Apache Spark, pySpark, and Databricks
• Develop and evolve cloud-native data solutions leveraging Azure and AWS services
• Model and optimize data structures for analytics, reporting, and downstream consumption
• Integrate data from multiple heterogeneous sources while ensuring data quality, consistency, and traceability
• Implement data quality controls, schema validation, lineage tracking, and monitoring processes
• Optimize performance, scalability, reliability, and cost-efficiency of data pipelines and infrastructure
• Collaborate closely with scientists, analysts, and engineering teams to translate business and research requirements into technical solutions
• Contribute to CI/CD processes, infrastructure automation, and engineering best practices
• Research and introduce new technologies and approaches within the data engineering ecosystem
• 3+ years of commercial experience in Data Engineering
• 2+ years of hands-on production experience with Databricks
• Strong expertise with Delta Lake, including medallion architecture and table optimization
• Advanced proficiency in Python, pySpark, and SQL
• Deep practical knowledge of Databricks components including Unity Catalog, Lakeflow Jobs / Workflows, and job optimization
• Hands-on experience with AWS services including S3, IAM, KMS, and at least one of AWS Batch, ECR, EventBridge, Lambda, or API Gateway
• Experience designing event-driven and asynchronous data pipelines
• Strong knowledge of Infrastructure as Code and CI/CD practices
• Practical experience with Terraform and Databricks Asset Bundles
• Experience working with Git workflows, pull request reviews, and automated testing
• Understanding of secure authentication approaches such as OIDC and service principals
• Practical experience with MLflow, experiment tracking, model registry, and reproducibility practices
• Strong understanding of data quality, lineage, schema contracts, and provenance concepts
• Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks DQX
• Experience monitoring and optimizing data platform reliability and cloud costs
• Upper-Intermediate or higher English level
WILL BE A PLUS:
• Experience packaging workloads using Docker and running batch workloads in AWS
• Familiarity with ECR and containerized data processing solutions
• Experience with experiment optimization libraries such as scikit-optimize, BoTorch, Ax, or scipy
• Understanding of ML feature engineering and leakage prevention techniques
• Experience working in regulated or high-security environments
• Knowledge of data governance, compliance, and data residency requirements
• Experience disabling or restricting AI-assisted tooling/features in secure environments
Skills
- AI
- Analytics
- API
- API Gateway
- Authentication
- Automation
- AWS
- Azure
- CI/CD
- Cloud
- Cloud Native
- Data Engineering
- Data Governance
- Data Pipelines
- Data Quality
- Databricks
- Delta Lake
- Docker
- Event Driven Architecture
- Eventbridge
- Feature Engineering
- Git
- IAM
- Infrastructure as Code
- Lambda
- Machine Learning
- Medallion Architecture
- MLflow
- OpenID
- PySpark
- Python
- S3
- scikit-learn
- SciPy
- Spark
- SQL
- Terraform
- Test Automation
- Unity
- Unity Catalog