Data Engineer: Scalable ETL, Airflow & Snowflake Expert
Posted Updated
Description
Endava is seeking a senior data engineer to design, build, and optimize data pipelines and architectures across cloud ecosystems. You will work with Python, SQL, and Airflow to deliver scalable data solutions and robust data quality checks.
The role involves collaboration with analytics and platform teams, cloud infrastructure, and CI/CD practices. The team emphasizes production-grade pipelines, observability, and incident response while supporting development and QA environments.
Design, develop, and maintain scalable ETL/ELT pipelines for data ingestion, transformation, validation, and delivery. Develop reusable, modular, and configuration-driven data processing solutions. Contribute to technical architecture and evaluate solutions for scalability, performance, reliability, and maintainability. Design and optimize data models, SQL queries, stored procedures, and data processing workflows. Design and maintain Apache Airflow workflows, including scheduling, dependencies, retries, monitoring, and error handling. Develop automated data validation and quality checks using DBT and SQL. Implement monitoring, alerting, and observability for critical data pipelines. Troubleshoot and resolve complex data processing and quality issues. Develop data solutions using AWS and cloud-based data warehouse technologies like Snowflake. Work with cloud services for data storage, processing, compute, messaging, and serverless workloads. Support data platforms across development, QA, staging, and production environments. Optimize data workloads for performance, scalability, reliability, and cost. Develop and maintain CI/CD pipelines for data engineering applications and infrastructure. Apply Infrastructure-as-Code practices using Terraform or equivalent technologies. Support production deployments, incident resolution, root-cause analysis, and continuous improvement. Maintain technical documentation, runbooks, and troubleshooting guides. Take ownership of complex engineering initiatives from design through production. Conduct code reviews and promote engineering best practices. Mentor junior and mid-level engineers and provide technical guidance. Collaborate with architecture, DevOps, product, and other technology teams. 5+ years of professional experience in data engineering or a related software engineering role. Strong hands-on experience building and supporting production-grade data pipelines. Strong knowledge of Apache Airflow, including DAG development, scheduling, dependencies, retries, and monitoring. Strong experience with Snowflake or equivalent cloud data warehouse technologies. Proven experience with DBT. Solid experience with AWS or another major cloud platform. Experience with CI/CD and Infrastructure-as-Code practices. Strong programming skills in Python and SQL. Demonstrated experience troubleshooting complex production issues and performing root-cause analysis. Ability to independently own complex technical initiatives from design through implementation and production. Strong communication, problem-solving, and technical documentation skills. Competitive salary Award recognition programs Referral bonus Career coaching Global career opportunities Non-linear career paths Management and technical leadership开发Skills
Sign in to read and join the discussion.