Data Engineer

Key Responsibilities

• Design and develop automated ELT processes ensuring a high level of data quality and reliability, using tools such as Spark, Apache Airflow, DBT, Python, Databricks, etc.

• Build and maintain batch and real-time data processing using Apache Kafka and AWS services like SNS, SQS, and Kinesis, supporting microservice architectures.

• Maintain the data lake in AWS and Databricks, building well-governed, query-oriented data models for efficient consumption.

• Implement monitoring and observability for the pipelines you work on.

• Collaborate cross-functionally, assisting other teams in identifying, collecting, and processing valuable information for diverse business use cases.

• Support the Data Science team by helping ensure stable, scalable environments for model deployment and experimentation.

• Build and maintain CICD pipelines, following best practices for automatic deployment and consistency.

• Help administer and maintain deployed infrastructure.

• Follow and contribute to engineering standards and best practices across the Data Engineering team.

• Deliver work on time, managing your own priorities in line with team commitments.

Skills and Experience Required

• 3+ years in data engineering or related data processing roles.

• Strong experience working with varied data formats and sources (JSON, CSV, Parquet, APIs, multiple DB engines).

• Hands-on experience with data modelling in Big Data platforms (e.g. Databricks, Glue, Snowflake).

• Familiarity with event-driven architectures and AWS.

• At least 2 years of hands-on experience with AWS, especially:

• Core: IAM, S3, EC2, VPC

• Data Services: EMR, Lambda, Batch, SNS, SQS, DynamoDB, Glue, Athena

• Ability to set up and maintain monitoring, alerting, and debugging solutions for your pipelines.

• Experience collaborating with Data Science teams to support production-grade pipelines.

Knowledge

• Solid understanding of computing systems (OS, memory, networks, etc.).

• Proficient in data analysis and interpretation, with a good grasp of statistics and data quality methods.

• Strong SQL and performance tuning skills.

• Good knowledge of Big Data technologies, Spark, and object storage systems.

• Proficiency in Python (and optionally Scala), with strong coding and testing practices.

• Familiarity with Docker; Kubernetes experience is a plus.

• Airflow (or similar) for orchestration.

• Good understanding of IaaC and CICD principles.

• Dashboard and storytelling experience for data visualization.

Other

• Strong communication skills.

• Proactive mindset and collaborative team spirit.

• Comfortable working within an agile team setup.