Cloud Data Engineer: Spark, Scala & Airflow on Kubernetes
Posted Updated
Description
Inetum is seeking a data engineer to migrate Hadoop to cloud-based data platforms and build scalable pipelines. You will transform data using Spark, Scala and SQL, implement testing, and automate deployments within an agile environment.
You will work with cross-functional teams to ensure data quality, governance, and operational excellence while contributing to knowledge sharing and continuous improvement in a dynamic, international setting.
Migrate Hadoop infrastructure to cloud infrastructure on Kubernetes Engine, COS, Spark as a service, and Airflow as a service. Implement data transformation and quality processes using Spark, Scala, and SQL. Set up CI/CD pipelines to automate deployments, unit tests and development management. Write and run unit and validation tests to ensure code accuracy and integrity. Automate data pipelines and orchestration with Airflow as a Service. Write technical documentation to capture specifications and operational details. Collaborate with cross-functional teams to meet data requirements and deliver solutions. Foster a culture of continuous learning and improvement within the team. Experience with Spark on Scala and Hadoop ecosystem. Strong SQL skills and knowledge of NoSQL databases. Hands-on data transformation, quality checks, and data enrichment tasks.Skills
Sign in to read and join the discussion.