Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Mindrift

Senior Python Data Scraping Engineer (Freelance)

Posted Updated

Description

Halo Labs in Australia is seeking a data engineer to design, develop, and maintain in-house PySpark frameworks and scalable ETL/ELT pipelines across Azure and AWS.

You will work with Python, PySpark, Databricks, and REST/SFTP data integrations, ensuring security, governance, and production reliability while collaborating with software teams to embed data services in the SaaS product.

Design and maintain reusable PySpark frameworks for the SaaS platform. Architect scalable ETL/ELT pipelines across Azure and AWS. Build distributed data processing with Python and PySpark on Databricks. Develop batch and near real-time ingestion pipelines from third-party systems. Design secure data integration patterns using REST, SFTP, and webhooks. Collaborate with software engineers to embed data services into the product. Implement CI/CD pipelines for automated deployment and testing. Apply infrastructure-as-code across Azure and AWS. Proficient in Python, SQL, and PySpark for large-scale data processing. Hands-on experience with Databricks and cloud data services (Azure/AWS). Experience building production data pipelines in SaaS environments. Strong understanding of cloud-native data architectures and governance. Learning budget Performance bonuses Wellbeing support

Skills

Apply

See also