Senior Data Engineer

The Ideal Candidate

Our client values logical thinkers who balance respect for best practices with independent critical thinking. They seek professionals who are adaptable, capable of owning projects from end-to-end, communicate effectively in English, and thrive in collaborative, high-performing team environments. While deep experience in specific technologies is valued, they prioritize candidates who are eager to learn, grow, and fill any gaps on the job.

Key Responsibilities

• Develop and Maintain Data Infrastructure: Build, optimize, and manage the data lake and its associated processing frameworks, ensuring reliable and scalable data ingestion from diverse sources.

• Design and Orchestrate ETL Pipelines: Architect and implement robust data workflows using orchestration tools, moving data from source systems through various processing stages into the lake.

• Enable Analytics and Data Science: Structure data schemas and design data models to support business intelligence, reporting, and machine learning initiatives.

• Integrate New Data Sources: Work with product and engineering teams to onboard new data streams, ensuring seamless integration and data quality.

• Champion Technology Evolution: Evaluate and introduce emerging tools and frameworks to continuously improve the data platform's performance and capabilities.

Required Experience & Skills

• Core Data Engineering Expertise: Solid understanding of data lake architectures and experience working with columnar big data databases (e.g., Athena, Redshift, Vertica, Hive/Hadoop). Familiarity with Iceberg is a significant advantage.

• ETL & Workflow Orchestration: Proven ability to design and implement ETL pipelines and manage workflows using tools such as Apache Airflow, Luigi, or AWS Batch.

• Programming Proficiency: Strong Python skills, with hands-on experience using relevant libraries (e.g., boto3, pandas, pytest). PySpark experience is highly desirable.

• Cloud & AWS Services: Practical experience with cloud environments—particularly AWS (Glue, EMR, EC2, S3, Lambda, IAM, CloudWatch) or equivalent platforms.

• Containerization & Orchestration: Familiarity with Docker and Kubernetes (or AWS ECS/EKS) for deployment and scaling.

• CI/CD & Version Control: Experience with CI/CD tools (e.g., CircleCI, Jenkins, AWS CodePipeline) and solid git practices (branching strategies, collaboration workflows).

• Agile Methodologies: Comfortable working within Agile/Lean frameworks such as Scrum or Kanban.

• Bonus Skills (Highly Valued):

  • Distributed messaging and streaming systems (Kafka, Pulsar, RabbitMQ)

  • Streaming processing frameworks (Spark Streaming, Apache Beam, Apache Flink)

  • Metadata catalogue and lineage tools (Amundsen, Apache Atlas, Alation)

  • JVM languages (Java, Scala, Kotlin) and related frameworks

  • RDBMS/NoSQL databases (PostgreSQL, MySQL, DynamoDB, Redis)

  • BI tools (Tableau, Looker, PowerBI, QuickSight)

  • Logging and monitoring stacks (ELK, Datadog, Prometheus, Grafana)

  • Data privacy and security concepts (encryption, tokenization, Apache Ranger)

Experience Level

This is a senior position (L3–L4 level). Candidates with approximately 3+ years of relevant data engineering experience are encouraged to apply. What matters most is the quality of experience—our client values professionals who have worked in environments that foster continuous learning and skill development. Length of tenure alone is not a differentiator; they seek individuals who can demonstrate meaningful growth and technical progression throughout their careers.