Senior Data Engineer (PYTHON, SQL) - HYBRID

Role responsibilities

• Design, develop and maintain data pipelines that extract, load and transform data from various sources into the data warehouse
• Design and implement data models that align with business requirements and support efficient querying and analysis
• Ensure the team’s orchestration pipelines are reliable, scalable and efficient whilst considering factors including data volume, velocity and variety
• Work across data engineering teams, vendors and internal stakeholders to contribute to the continuous improvement of data products, processes and systems.
• Follow data engineering best practices including coding, DevOps, documentation and metadata standardizations
• Experience with the management and nature of audience tracking and subscribers’ datasets

Employee responsibilities

• Take ownership and accountability for assigned work, including development and end-to-end testing
• Ability to conduct comprehensive feasibility assessments of architectural and data modelling options, systematically evaluating the trade-offs, advantages, and limitations of each proposed solution
• Ensure high standard and ongoing quality of Engineering output whilst meeting project timelines
• Document existing and future solution designs to enhance the team's collective knowledge

Qualifications, Experience & Skills

Essential qualifications, experience & skills

• Fluent in Python programming language
• Proficient in SQL
• Understanding of OLTP concepts including normalised schema design
• Understanding of OLAP concepts and data model design
• Experience leading the designing and building data pipelines, ETL processes, and managing data infrastructure
• Experience leading workflows automation, managing task dependencies and transformations using Apache Airflow and DBT
• Experience in real time streaming pipelines like Kafka/pubsub and Pyspark/dataflows

Problem-Solving & Leadership:

• A strong analytical mindset, a passion for resolving complex technical challenges, and the ability to find creative solutions to issues
• Excellent communication and interpersonal skills, with the ability to lead technical discussions, build consensus, and work effectively with cross-functional teams
• Deep understanding of cloud migration strategies, best practices, and potential pitfalls

Desirable qualifications, experience & skills

• A bachelor's or master's degree in computer science, software engineering, data engineering, data science, information technology or a related field
• 5+ YRS in data engineering
• Experience with Google Cloud Platform, AWS or Azure including services for data storage, processing, and analytics
• Experience with cloud deployment and test automation using a CI/CD solution such as Cloud Build or Concourse
• Familiarity with consumer behaviour data (Google Analytics, Adobe Analytics or Snowplow Analytics) and/or subscribers’ data
• Proficient usage of AI to gain working efficiency to focus on higher impact architectural tasks

Key Business Relationships Interactions

Internal Stakeholders

• Collaborate with adjacent technology teams including cyber security, network, and data platform engineering to build and maintain data products that meet organisational standards and policies, and respond to and resolve incidents
• Partner with product managers to shape data products and solutions, understand requirements and support end users
• Support end-users through knowledge sharing and issue investigation

External Stakeholders

• Engage appropriate vendor support channels to resolve issues with third-party technology