Senior Data Engineer—Python & Spark, Real-Time Pipelines
Summary
Build and optimize real-time data pipelines in Python and Spark to power trading research at a global market maker.
Senior Data Engineer - Python & Spark, Real-Time Pipelines at Optiver Private Jobs.
About the role Optiver Sydney is on the lookout for a seasoned Senior Data Engineer to enhance the data capabilities of our research team. You will collaborate with local researchers and our Global Research and Data Platform teams to implement workflows, improve large-scale data processing, and effectively manage infrastructure expenses.
Key facts Location: Australia Engagement: Full-time Compensation: $180,000 - $260,000
What you'll do
- Collaborate with researchers to transition workflows into production environments.
- Optimize and maintain large-scale data processing systems.
- Monitor and control infrastructure costs related to data operations.
- Work with cross-functional teams to ensure data solutions meet business needs.
Requirements
- A minimum of 5 years of experience in Data Engineering or a related field.
- Proficiency in Python and Apache Spark for data processing.
- Strong understanding of data modeling and ETL processes.
- Experience with cloud platforms, preferably AWS or Azure.
- Familiarity with SQL and NoSQL databases.
- Ability to work effectively in a team-oriented environment.
Nice to have
- Experience with real-time data processing frameworks.
- Knowledge of machine learning concepts and tools.
- Familiarity with containerization technologies such as Docker or Kubernetes.
Skills & tools
- Python
- Apache Spark
- SQL and NoSQL databases
- Cloud services (AWS, Azure)
- Data modeling and ETL tools
Practical notes
- Visa sponsorship is available for qualified candidates.
- Competitive benefits package including health insurance and retirement plans.
- Applications will be accepted until (insert application deadline).
