Databricks engineer (AWS)

Mandate skills

Databricks + AWS ( Databricks + AWS Data Engineer, with strong hands-on expertise in Databricks, PySpark/SQL, and AWS data services)

AWS Databricks

Role Summary

We are looking for an experienced Databricks + AWS Data Engineer with strong hands-on expertise in Databricks, PySpark, SQL and AWS data services. The candidate will be responsible for designing, developing and optimising scalable data pipelines and data processing solutions on the AWS cloud platform.

Key Responsibilities

• Design, develop and maintain scalable ETL/ELT data pipelines using Databricks.
• Develop large-scale data processing solutions using PySpark and Spark SQL.
• Build and integrate data solutions using AWS services such as Amazon S3, AWS Glue, Amazon Redshift, AWS Lambda and IAM.
• Perform data ingestion, transformation, cleansing and integration across multiple data sources.
• Work with Delta Lake and Databricks Lakehouse architecture for reliable and scalable data processing.
• Optimise Spark workloads, SQL queries and Databricks jobs for performance and cost efficiency.
• Implement appropriate data quality, monitoring and error-handling mechanisms.
• Troubleshoot production data pipeline and performance issues.
• Collaborate with data architects, analysts and business teams to translate requirements into technical solutions.
• Follow coding, security, version-control and deployment best practices.

Mandatory Skills

• Strong hands-on experience with Databricks for data engineering and large-scale data processing.
• Strong expertise in PySpark / Apache Spark for developing and optimising data pipelines.
• Advanced proficiency in SQL, including complex queries and performance optimisation.
• Hands-on experience with AWS data services, particularly Amazon S3 and AWS Glue.
• Strong experience in designing and developing ETL/ELT pipelines and data transformations.
• Working knowledge of Delta Lake and Lakehouse architecture.
• Proficiency in Python for data engineering and automation.
• Good understanding of data modelling, data integration and data quality concepts.