Senior Data Engineer
Summary
Senior Data Engineer at ShyftLabs designs and builds enterprise-scale data platforms for Fortune 500 clients using Databricks, Spark, Python, and cloud platforms, while leading teams and client engagements.
What You'll Be Doing
- Lead the architecture, design, and implementation of enterprise-scale data platforms from project inception through production deployment.
- Own technical delivery across multiple client engagements while ensuring high-quality engineering standards.
- Define solution architecture, technical roadmaps, and implementation strategies aligned with client business goals.
- Conduct architecture reviews, code reviews, and establish engineering best practices across project teams.
- Mentor and coach Data Engineers while fostering technical excellence and continuous learning.
- Serve as the primary technical leader for complex engineering initiatives and critical project decisions.
- Partner directly with Fortune 500 clients to understand business requirements and translate them into scalable technical solutions.
- Lead discovery workshops, architecture sessions, and technical planning meetings with both business and engineering stakeholders.
- Present solution designs, delivery plans, and architectural recommendations to technical leadership and executive audiences.
- Build trusted relationships with client teams while providing technical guidance throughout project execution.
- Support pre-sales activities by contributing technical expertise, solution estimates, and implementation approaches when required.
- Design, develop, and optimize enterprise-grade data pipelines using the Databricks Unified Analytics Platform.
- Build scalable ETL and ELT frameworks capable of processing large-scale structured and unstructured datasets.
- Design and implement Lakehouse architectures using Delta Lake and Medallion design patterns.
- Develop high-performance Spark applications for batch and real-time data processing.
- Integrate data from enterprise applications, APIs, streaming platforms, and cloud storage solutions.
- Ensure data quality, integrity, and reliability through automated validation, testing, and monitoring.
- Architect cloud-native data platforms across AWS, Azure, or Google Cloud Platform.
- Implement Infrastructure-as-Code using Terraform or similar technologies.
- Build and maintain CI/CD pipelines supporting automated testing and deployment.
- Optimize cloud infrastructure for scalability, reliability, security, and cost efficiency.
- Monitor platform performance and proactively resolve operational issues.
- Implement enterprise data governance frameworks and security best practices.
- Configure Unity Catalog, metadata management, lineage, and role-based access controls.
- Ensure compliance with organizational security standards and regulatory requirements.
- Promote data observability and operational excellence across production environments
- Partner closely with Product Managers, Data Scientists, Analytics Engineers, Machine Learning Engineers, and Software Engineers to deliver high-impact data products.
- Enable AI and machine learning initiatives through scalable feature engineering pipelines and production-ready datasets.
- Contribute reusable frameworks, accelerators, and engineering standards that improve delivery across client engagements.
What You'll Bring
- Bachelor's or Master's degree in Computer Science, Data Engineering, Software Engineering, or a related technical discipline.
- 8+ years of experience designing and building enterprise-scale data platforms.
- 5+ years of hands-on experience with Databricks and Apache Spark.
- Proven experience leading enterprise data engineering projects from architecture through production delivery.
- Strong expertise in Python, SQL, and Spark for large-scale data processing.
- Deep understanding of Delta Lake, Lakehouse architecture, and modern data platform design.
- Experience working with AWS, Azure, or Google Cloud Platform.
- Strong knowledge of ETL/ELT frameworks, distributed computing, and data modeling.
- Experience implementing CI/CD pipelines and Infrastructure-as-Code.
- Strong understanding of data governance, security, metadata management, and data quality practices.
- Experience optimizing distributed data processing workloads for performance and cost.
- Excellent communication and stakeholder management skills with experience working directly with enterprise clients.
- Demonstrated ability to mentor engineers and lead technical initiatives
Nice to Have
- Databricks Certified Professional Data Engineer certification.
- Experience with Delta Live Tables, MLflow, Unity Catalog, and Databricks SQL.
- Experience with Kafka, Kinesis, Event Hubs, or other streaming technologies.
- Hands-on experience with Snowflake, dbt, Airflow, or modern data orchestration tools.
- Experience with Kubernetes, Docker, and Terraform.
- Knowledge of AI/ML data platforms, Feature Stores, or Retrieval-Augmented Generation (RAG) architectures.
- Previous consulting or professional services experience delivering solutions for enterprise clients.
- Experience within retail, e-commerce, financial services, logistics, healthcare, or ad-tech environments.
Salary Range
- $140,000 – $180,000 (CAD)
Skills
- AI
- Airflow
- Analytics
- API
- AWS
- Azure
- CAD
- CI/CD
- Cloud
- Cloud Native
- Data Engineering
- Data Governance
- Data Modeling
- Data Pipelines
- Data Quality
- Databricks
- dbt
- Delta Lake
- Design Patterns
- DevOps
- Distributed Computing
- Docker
- E-commerce
- ELT
- ETL
- Feature Engineering
- GCP
- Infrastructure as Code
- Kafka
- Kinesis
- Kubernetes
- Lakehouse
- Machine Learning
- Metadata Management
- MLflow
- Observability
- Pre-sales
- Python
- RAG
- Snowflake
- Spark
- SQL
- Stakeholder Management
- Terraform
- Test Automation
- Unity
As published by lever · 3 questions
Basics
Resume/CV, Full name, Email, Phone, Current location, Current company, LinkedIn URL, GitHub URL, Portfolio URL
Short answers (1)
- How soon can you join after receiving an offer?
Pick from a list (2)
- Are you authorized to work for any employer in Canada? optional
- Will you require sponsorship now or in the future? optional