Senior Data Engineer – Remote, Flexible Hours
Posted Updated
Description
AgileEngine is seeking a Lead Data Engineer to own the data pipeline and analytical architecture for a high-volume marketing analytics platform. You will decide partitioning strategies, file formats, and near-real-time processing for OLAP workloads on an S3-backed data lake, and design ETL pipelines and Airflow DAGs.
You will lead senior developers and collaborate with DevOps to align on requirements. The role demands autonomy, strategic thinking, and strong AWS and Python skills, with fluency
Design and own ETL pipelines at scale, integrating internal and external data sources. Make architectural calls on partitioning, file formats, and near-real-time processing for OLAP workloads. Own batch workflow design with Airflow, including structure, dependencies, and triggering strategy. Drive use of AWS data stack and partner with DevOps to meet requirements. Review PRs and enforce coding standards; guide senior developers and ensure architecture alignment. 7+ years of engineering experience with ETL and large-volume data systems. Hands-on OLAP architecture experience with tools like Athena/Trino/Presto, BigQuery, or similar. Experience designing data lakes on object storage (S3 or equivalent) with partitioning and Parquet/ORC formats. Deep familiarity with DAG-style workflow orchestration, preferably Airflow; others are acceptable. Proficiency across the AWS data stack (S3, Athena, EKS) and collaboration with DevOps. Backend Python development and REST/GraphQL interfaces. Strong stance on code quality, testing, and AI-assisted development with responsible use. Learning budget Competitive compensation Remote work with flexible hours Modern, impactful projects Collaborative culture Well-being programsSkills
Sign in to read and join the discussion.