Data Platform Engineer
Posted 16
views
Summary
Data Platform Engineer at Shift4 in Toronto: build, tune, and scale AWS ECS/SQS-based pipelines replicating Aurora MySQL data into Snowflake, owning reliability, SLOs, observability, cost efficiency, and IaC/CI-CD for the data platform infrastructure.
Shift4 (NYSE: FOUR) is boldly redefining commerce by simplifying complex payments ecosystems across the world. As the leader in commerce-enabling technology, Shift4 powers billions of transactions annually for hundreds of thousands of businesses in virtually every industry. For more information, visit www.shift4.com.
Shift4 is seeking a
Data Platform Engineer
to join our Data Platform team and build, maintain, and optimize the pipelines that move data from our production systems into our analytics warehouse. Today, our infrastructure uses Amazon ECS services and SQS queues to replicate data from Aurora MySQL into Snowflake, with plans to expand to additional data sources and consumers as our platform evolves.
This is a hands‑on infrastructure engineering role focused on pipeline performance, reliability, scalability, resource optimization, cost efficiency, and observability. You will help ensure data is delivered accurately, reliably, and on time as our data platform grows. Data modeling, warehouse cost management, and business transformation logic are managed by partner teams, allowing you to focus on the infrastructure and systems that power our data ecosystem—all while embracing the Shift4 way.
Responsibilities
Operate, tune, and scale the ECS/SQS-based ingestion pipelines from Aurora MySQL into Snowflake: task sizing, autoscaling, consumer concurrency, batching, retries, and dead‑letter handling
Own pipeline reliability: define SLOs for freshness and lag, build alerting and dashboards, lead incident response and root‑cause analysis for pipeline failures
Build and maintain infrastructure as code, deployment automation, and CI/CD for pipeline services
Continuously benchmark and tune throughput versus cost across ECS, SQS, and Snowflake ingestion (Snowpipe/COPY, stages)
Onboard new sources and consumers to the platform in partnership with application, data, and analytics teams
Apply data‑quality and security controls appropriate for PCI‑compliant or payment‑handling environments: encryption, secrets handling, least‑privilege access, audit trails
Proactively explore and apply AI tools to improve efficiency across the pipeline lifecycle (development, testing, runbooks, and documentation)
Qualifications
5+ years in platform, infrastructure, or DevOps roles supporting data‑intensive systems in production
Deep hands‑on AWS experience: ECS/Fargate (task sizing, autoscaling, deployments), SQS (throughput, DLQs, visibility timeouts, batching), S3, IAM, CloudWatch, and Aurora/RDS from the operator side
Proven track record tuning message‑driven workloads: consumer concurrency, backpressure, retries, idempotency, throughput versus cost
Experience operating pipelines into Snowflake from the ingestion side (Snowpipe/COPY, stages, load monitoring)
Infrastructure as code with Terraform/Ansible, containerization with Docker, and CI/CD pipelines
Observability and alerting for pipeline health and lag using Prometheus/Grafana, CloudWatch, Datadog, or the Elastic Stack
Enough Python to read, debug, and instrument worker code
Passion for automation, performance, and clean system design
Nice to have
CDC tooling (Debezium, AWS DMS) and MySQL Binlog replication
Message brokers beyond SQS: Apache Kafka/Confluent, RabbitMQ
Elastic Stack or time‑series database (Prometheus, InfluxDB) administration
Experience in a PCI or otherwise regulated environment
Payments or fintech domain knowledge
We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race, color, sex, age, national origin, religion, sexual orientation, gender identity and/or expression, status as a veteran, and basis of disability or any other federal, state or local protected class.
Shift4 is seeking a
Data Platform Engineer
to join our Data Platform team and build, maintain, and optimize the pipelines that move data from our production systems into our analytics warehouse. Today, our infrastructure uses Amazon ECS services and SQS queues to replicate data from Aurora MySQL into Snowflake, with plans to expand to additional data sources and consumers as our platform evolves.
This is a hands‑on infrastructure engineering role focused on pipeline performance, reliability, scalability, resource optimization, cost efficiency, and observability. You will help ensure data is delivered accurately, reliably, and on time as our data platform grows. Data modeling, warehouse cost management, and business transformation logic are managed by partner teams, allowing you to focus on the infrastructure and systems that power our data ecosystem—all while embracing the Shift4 way.
Responsibilities
Operate, tune, and scale the ECS/SQS-based ingestion pipelines from Aurora MySQL into Snowflake: task sizing, autoscaling, consumer concurrency, batching, retries, and dead‑letter handling
Own pipeline reliability: define SLOs for freshness and lag, build alerting and dashboards, lead incident response and root‑cause analysis for pipeline failures
Build and maintain infrastructure as code, deployment automation, and CI/CD for pipeline services
Continuously benchmark and tune throughput versus cost across ECS, SQS, and Snowflake ingestion (Snowpipe/COPY, stages)
Onboard new sources and consumers to the platform in partnership with application, data, and analytics teams
Apply data‑quality and security controls appropriate for PCI‑compliant or payment‑handling environments: encryption, secrets handling, least‑privilege access, audit trails
Proactively explore and apply AI tools to improve efficiency across the pipeline lifecycle (development, testing, runbooks, and documentation)
Qualifications
5+ years in platform, infrastructure, or DevOps roles supporting data‑intensive systems in production
Deep hands‑on AWS experience: ECS/Fargate (task sizing, autoscaling, deployments), SQS (throughput, DLQs, visibility timeouts, batching), S3, IAM, CloudWatch, and Aurora/RDS from the operator side
Proven track record tuning message‑driven workloads: consumer concurrency, backpressure, retries, idempotency, throughput versus cost
Experience operating pipelines into Snowflake from the ingestion side (Snowpipe/COPY, stages, load monitoring)
Infrastructure as code with Terraform/Ansible, containerization with Docker, and CI/CD pipelines
Observability and alerting for pipeline health and lag using Prometheus/Grafana, CloudWatch, Datadog, or the Elastic Stack
Enough Python to read, debug, and instrument worker code
Passion for automation, performance, and clean system design
Nice to have
CDC tooling (Debezium, AWS DMS) and MySQL Binlog replication
Message brokers beyond SQS: Apache Kafka/Confluent, RabbitMQ
Elastic Stack or time‑series database (Prometheus, InfluxDB) administration
Experience in a PCI or otherwise regulated environment
Payments or fintech domain knowledge
We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race, color, sex, age, national origin, religion, sexual orientation, gender identity and/or expression, status as a veteran, and basis of disability or any other federal, state or local protected class.
