Data Engineer GCP
Posted 9
views
Summary
DS STREAM is hiring a Data Engineer (remote, Warsaw-based) to build, scale, and maintain production-scale data platforms, including real-time pipelines, ML infrastructure, and attribution data processing. Core stack: GCP with Python, SQL, Apache Spark, Airflow, BigQuery, dbt, plus Terraform, Docker, and Kubernetes.
Data Engineer
We’re looking for an experienced Data Engineer to build, scale, optimize, and maintain reliable data platforms. You’ll work on real-time pipelines, ML infrastructure, attribution data processing, and cloud-native solutions in a high-volume production environment.
Tech Stack
- GCP
- Apache Spark, dbt, BigQuery
- Python, SQL, Scala
- Apache Airflow
- Terraform, GitHub Actions, Docker, Kubernetes
- VertexAI
Must Have
- Over 3 years of experience in Data Engineering and building production-scale data platforms.
- Strong programming skills in Python and SQL.
- Advanced experience with data modeling, ETL development, and multiple data formats.
- Strong knowledge of Google Cloud Platform and cloud-native architecture design.
- Expert knowledge of Apache Airflow for workflow orchestration and automation.
- Hands-on experience with Apache Spark for batch and high-volume streaming workloads.
- Good understanding of CI/CD and DevOps tools such as GitHub Actions or Kubernetes.
Nice to Have
- Experience with MLOps and ML deployment in production, preferably with Vertex AI.
- Practical experience with Terraform for Infrastructure as Code.
- Experience building scalable REST APIs for data or ML services.
- Strong testing practices, including TDD and unit/integration tests for Spark and Airflow.
- Knowledge of Scala for high-performance data processing.