Software Engineer - Cloud Data Pipelines (R/Python)
Summary
Design, build, and maintain cloud data pipelines that ingest, process, and run custom R and Python analytical scripts at scale on GCP or Azure. Day-to-day involves containerizing workloads, orchestrating workflows with tools like Airflow, and integrating models into production using managed cloud services.
Permanent/Fixed term Full time Employment
We are seeking a Software Engineer to design, build, and maintain robust data pipelines that ingest, process, and execute custom R and Python scripts at scale. You will work within our cloud infrastructure (Google Cloud Platform or Azure) to automate data workflows, ensure high performance, and integrate advanced analytical models into production environments.
Key Responsibilities
- Execute R/Python Workloads: Containerize, deploy, and orchestrate custom R and Python scripts to run reliably in cloud-native serverless or containerized environments.
- Cloud Integration: Leverage managed cloud services on GCP (e.g., Cloud Functions, Cloud Run, BigQuery, Dataflow) or Azure (e.g., Azure Data Factory, Azure Functions, Databricks, Synapse Analytics).
- Workflow Orchestration: Implement and maintain workflow tools (such as Apache Airflow or Prefect) to schedule and monitor script execution.
- Ensure Data Quality: Implement automated logging, error handling, and testing frameworks to validate data integrity and script outputs.
- Optimize Performance: Tune data processing jobs and cloud resource allocation for cost-efficiency, speed, and scalability.
Required Qualifications & Technical Skills
- Programming: Advanced proficiency in Python and/or R, with strong experience writing clean, modular, and well-tested code.
- Cloud Platforms: Hands-on production experience with either Google Cloud Platform (GCP) or Microsoft Azure.
- Containerization & Tools: Experience with Docker, Kubernetes, and CI/CD pipelines (GitHub Actions, GitLab CI).
- Data Stores: Proficiency with SQL and relational/non-relational databases (PostgreSQL, BigQuery, Snowflake, or Azure SQL).
- Software Engineering Practices: Solid foundation in version control (Git), automated testing, and API design.
Preferred Qualifications
- Experience migrating legacy local R/Python analytical scripts into distributed cloud architectures.
- Knowledge of Infrastructure as Code (Terraform or Bicep).
Skills
- Airflow
- Analytics
- API
- Api Design
- Azure
- Azure Data Factory
- Azure Functions
- Azure Synapse
- Bicep
- BigQuery
- CI/CD
- Cloud
- Cloud Native
- Containerization
- Data Pipelines
- Data Quality
- Databricks
- Docker
- GCP
- Git
- GitHub
- GitHub Actions
- GitLab
- Infrastructure as Code
- Kubernetes
- PostgreSQL
- Prefect
- Python
- Serverless
- Snowflake
- SQL
- Terraform
- Test Automation
- Version Control
- Workflow Orchestration