Senior Data Engineer - Databricks/Spark
Summary
Senior Data Engineer building and maintaining large-scale Databricks/Spark data pipelines (Delta Live Tables, Unity Catalog, Delta Sharing, SQL Warehouse) orchestrated with Airflow and GitHub Actions CI/CD. Day-to-day covers serverless migration of pipelines, CI/CD ownership, code review, and writing Python pipeline code for a pricing/financial data domain.
Requirements:
- Bachelor's or Master's degree in Computer Science, Engineering or a related field
- 8+ years of experience in Data Engineering with strong big data expertise
- Deep expertise in Databricks (2+ years of hands-on experience), Spark and SQL (including Spark SQL)
- Strong data modeling and ETL experience with large-scale datasets
- Hands-on experience orchestrating data pipelines with Airflow and CI/CD (GitHub Actions or similar)
- Proven experience with Delta Live Tables, Unity Catalog, Delta Sharing and SQL Warehouse (server and serverless)
- Proficiency in Python for data engineering workflows
- Familiarity with analytics engineering and dbt
- Experience with data governance, compliance and security in a cloud environment
- Experience adopting software engineering best practices (testing, modularization, refactoring, automation)
- Ability to optimize data systems for performance and cost efficiency in cloud environments
- Exposure to AI/ML frameworks for development efficiency
- Demonstrated ability to lead projects independently and mentor junior engineers
- English: Upper-Intermediate or higher
Nice to have:
- Experience with serverless architectures
- Understanding of pricing / financial data domains
- Experience with cascading formula-based financial calculations
- Familiarity with enterprise client support workflows
Soft Skills:
- Strong self-management: ability to work independently and meet deadlines
- Proactive and ownership-driven mindset
- Effective and clear communication, including in distributed teams
- Collaborative, open to feedback, and focused on shared goals
- Strong problem-solving skills, especially in ambiguous contexts
- High attention to detail and code quality
- Adaptability to shifting priorities and new technologies
Responsibilities:
- Serverless migration: adapt existing pipelines / code to run on different cluster types (serverless) without issues
- CI/CD ownership: maintain and improve CI/CD processes, propose architectural improvements
- Pipeline monitoring: proactively check repositories, review merge requests and team members' changes
- Functional and technical work: write pipeline code and understand the business logic (pricing domain)