Senior Data Platform Engineer (Databricks)

We are looking for a seasoned Senior Data Platform Engineer to join EPAM, a company at the forefront of shaping digital solutions for Fortune 1000 businesses.

As a critical part of our team, the ideal candidate will bring deep expertise in Databricks and experience across multi-cloud environments. This role will focus on building, optimizing, and maintaining innovative, high-performance data platforms that enable powerful data insights and drive meaningful business transformations.

This position offers a hybrid work setup with flexibility to work from Malaga or Madrid, with occasional office visits required.

Responsibilities

• Architect and deploy robust data platforms using Databricks, focusing on optimal performance and security
• Create solutions that are cloud-agnostic across AWS, Azure, and GCP to ensure system flexibility and resilience
• Design and implement comprehensive data pipelines involving data lakes, warehouses, and streaming technologies
• Utilize Databricks SQL, Delta Lake, MLflow, and Spark for data interaction and performance enhancements
• Collaborate with various teams to implement and maintain workflows based on Databricks best practices
• Develop CI/CD pipelines tailored for data platform deployment and testing
• Set up and manage frameworks for monitoring, logging, and alerting to ensure infrastructure health
• Optimize compute and storage resources to balance cost-efficiency and performance
• Troubleshoot Databricks and Spark performance issues
• Mentor team members on effective cluster management and resource allocation in Databricks environments
• Maintain compliance and security standards throughout platform operations
• Drive adoption of advanced Databricks capabilities like Photon and Graviton instances
• Regularly refine architectures to align with evolving business and technology requirements

Requirements

• Extensive experience in Databricks, Apache Spark, and distributed data processing systems
• Strong programming skills in Python, Scala, SQL
• Proficiency in data engineering services on AWS (S3, IAM, Lambda), Azure, GCP
• Expertise in data architecture and ETL workflows, focusing on data lakes and lakehouses
• Hands-on experience with Terraform, CloudFormation, and CI/CD tools
• Familiarity with monitoring tools and observability frameworks for large-scale data environments
• Solid communication skills in Spanish (at least C1)

Nice to have

• Certifications in Databricks, AWS, Azure, GCP
• Knowledge of Kubernetes and containerized deployments for data pipelines
• Experience with real-time data streaming frameworks and governance tools

Benefits

• Private health insurance
• EPAM Employees Stock Purchase Plan
• 100% paid sick leave
• Referral Program
• Professional certification
• Language courses