Senior Data Engineer - Databricks & Real-Time Streaming
Summary
A senior data engineer building and maintaining data processing solutions on Databricks with Python, and designing real-time streaming pipelines using Apache Spark Structured Streaming. Day to day involves troubleshooting and optimizing streaming jobs for large data volumes and collaborating with engineers and stakeholders.
PERSOL Singapore PTE. LTD. is seeking an experienced Data Engineer to develop and maintain data processing solutions using Databricks and Python, and to design real-time streaming pipelines with Apache Spark Structured Streaming.
You will troubleshoot and optimize streaming jobs, work with large data volumes, and collaborate with data engineers and stakeholders to deliver effective solutions while following best practices for performance and maintainability.