Data Engineer |Databricks | 12 month agency contract
Posted 5
views
Summary
12-month agency contract role in Singapore for a data engineer who builds and maintains Databricks-based data processing solutions in Python, designs real-time streaming pipelines with Apache Spark Structured Streaming, and troubleshoots and optimizes large-scale data workflows.
Key Responsibilities:
- Develop and maintain data processing solutions using Databricks and Python.
- Design and implement real-time data processing pipelines using Apache Spark Structured Streaming.
- Develop efficient and scalable data processing logic using strong Python programming fundamentals and data structures.
- Troubleshoot and optimize streaming jobs and data processing workflows.
- Work with large volumes of data and ensure reliable, scalable, and efficient processing.
- Monitor and resolve issues related to streaming pipelines and Databricks workloads.
- Collaborate with data engineers, technical teams,and stakeholders to understand requirements and deliver effective solutions.
- Follow best practices for code quality,performance, maintainability, and operational support.
- Participate in troubleshooting, debugging, and performance optimization of data pipelines.
Required Skills & Experience:
- 5–10 years of overall experience in data engineering / data development.
- Strong hands-on experience with Databricks.
- Strong experience with ApacheSpark Structured Streaming.
- Strong Pythonprogramming skills with a good understanding of:
- Experience developing and troubleshooting batch and/or streaming data pipelines.
- Good understanding of data processing concepts anddistributed computing.
- Strong problem-solving and analytical skills.
- Ability to work independently as well as collaboratively with technical teams.
We regret that only shortlisted candidates will be notified.