Data Engineer

About Shop2Shop

The ability to make a payment quickly and at a low cost is an underestimated bedrock of our economy in South Africa. Payments are the financial lifeline of every business and individual. We are on a mission to help entrepreneurs participate and prosper in the digital economy by solving their payment challenges. When you join Shop2Shop, you join a team of people driven by a shared purpose for why we do what we do every day. The experience of collaboration across teams will be unrivalled. The feeling of belonging to a talented team of people will make this your last career stop. You will find a stimulating and challenging smorgasbord of problems to solve with the aim of supporting entrepreneurs in South Africa.

Purpose of role:

Every decision Shop2Shop makes about merchants, risk and growth depends on data arriving on time and being right. The Data Engineer builds and runs the pipelines that make that possible.

You will design, build and maintain Shop2Shop’s modern data platform, using Python and Dagster to develop data pipelines centred on software-defined assets. The role applies modern software engineering practices to data infrastructure, with a focus on testability, data lineage, declarative asset management and reliable data delivery.

Reporting to the Core Data Services Manager, you will work closely with data analysts, DevOps and business stakeholders to turn data requirements into scalable technical solutions.

Success = The people who rely on our data can trust it without checking it twice.

You’ll thrive in this role if…

• You approach data infrastructure with a software engineering mindset and value clean, maintainable Python code.
• You care about testability, data lineage and declarative asset management.
• You take ownership of data quality, monitoring and the reliability of data pipelines.
• You enjoy automating deployments and connecting tools across a modern data stack.
• You work well with data analysts, DevOps teams and business stakeholders to translate data requirements into scalable solutions.
• You are curious about how AI tools can make you faster and better, and you check what they produce.

What You’ll Do (and what success looks like)

Dagster Assets and Data Lineage

• Design, build and maintain robust, declarative software-defined assets and graphs in Dagster
• Use Dagster to track data lineage and state across the data stack

Success = Anyone can see where a number came from and how fresh it is.

Pipeline Development

• Write clean, efficient and maintainable Python code to extract, transform and load data from disparate sources into the data warehouse
• Use ETL or ELT processes as best fits each source
• Build data models in line with the architecture and modelling standards set for the platform

Success = Data lands in the warehouse on time, every time, without manual fixes.

Data Quality and Observability

• Implement rigorous data validation, testing and monitoring using tools such as Pytest and Dagster’s built-in features
• Make sure data quality issues are detected and flagged before they reach reports and dashboards

Success = Data problems are caught by our checks before someone finds them in a dashboard.

Production Support

• Own the day-to-day running of the pipelines you build, including failures, reruns and backfills
• Investigate and resolve pipeline incidents, and fix the root cause so they do not recur
• Keep runbooks and documentation current so others can support the platform

Success = When a pipeline fails, it is noticed, fixed and understood quickly, and the same failure does not happen twice.

Infrastructure and Integration

• Connect Dagster with the broader modern data stack, including dlthub, dbt, DuckDB, Athena, PostgreSQL and Terraform
• Integrate the data platform with AWS cloud infrastructure, working with DevOps, who own the underlying platform and clusters

Success = The data stack works as one platform rather than a set of separate tools.

CI/CD and Deployment

• Automate the deployment of Dagster code and data infrastructure using Docker, Kubernetes and GitLab CI
• Work within the platform and infrastructure standards set by DevOps

Success = Pipeline changes ship safely and repeatably, with no manual deployment steps.

Data Protection

• Handle merchant, customer and transaction data in line with POPIA and Shop2Shop’s data governance standards
• Apply the access controls, masking and retention rules defined in those standards
• Flag data protection risks early

Success = Personal data is protected by how the pipelines are built, not by remembering to be careful.

Use AI to Improve Delivery

• Use approved AI tools to speed up coding, testing and documentation
• Review AI-generated code with the same rigour as any other code before it ships
• Share what works with the Data team

Success = AI makes the team faster without lowering the quality bar.

Collaboration

• Partner with data analysts, DevOps teams and business stakeholders to understand data requirements
• Translate those requirements into scalable technical solutions

Success = Analysts and the business get data in the shape they need without long back and forth.

Core Competencies

• Systems Thinking – Understands dependencies and sees the bigger picture
• Problem Solving – Solves issues practically and quickly. Balances short-term fixes with long-term improvement.
• Collaboration – Works well across technology teams. Supports shared success over silo thinking.
• Continuous Improvement – Looks for better ways of working. Embraces automation, AI, and smarter delivery methods.
• Curiosity and critical thinking – Asking “why” and “what if” before jumping to “how”
• Communication – You translate technical issues into language business stakeholders understand.

Skills & Knowledge

• Strong, idiomatic Python, including ETL and ELT pipeline design
• Dagster concepts, including Software-Defined Assets, Ops, Graphs, Resources and I/O Managers, or the equivalent concepts in a similar orchestration tool such as Airflow or Prefect
• Strong SQL and cloud data warehouses such as Athena, Snowflake, BigQuery or Redshift
• Data modelling approaches such as Kimball dimensional modelling, Data Vault or one big table, and how to apply them to data platform requirements
• Data validation and testing with tools such as Pytest
• Git, testing frameworks, code reviews and CI/CD pipelines, applied with discipline
• Practical use of approved AI-assisted development tools, with careful review of what they produce
• Awareness of data privacy and POPIA requirements when handling personal and transaction data
• dbt, including integrating dbt projects into Dagster (advantageous)
• Containerisation and orchestration with Docker and Kubernetes (advantageous)
• Infrastructure as code with Terraform (advantageous)

Experience & Qualifications

• 3–5 years of professional software engineering or data engineering experience
• Experience building, deploying and supporting production data pipelines with Dagster or a similar orchestration tool, on a cloud data warehouse
• Relevant IT qualification or equivalent practical experience

Growth Mindset

At Shop2Shop, we believe growth is part of the job, not just for the company, but for every individual. In this role, you are expected to:

• Embrace continuous learning and stay curious about your craft
• Seek feedback and reflect on both successes and setbacks
• Set personal development goals and take ownership of your growth
• Support others in their growth journeys through knowledge sharing and collaboration
• Stay open to change and adapt quickly in a fast-paced environment

This role is for someone who takes pride in their craft and wants to build data that people can trust without a second thought.

Other Requirements

• Hybrid working model with regular in-office presence in Stellenbosch based on team rhythms, collaboration needs, and business requirements
• Ability to be available for key planning sessions, stakeholder workshops, and in-person team collaboration when required