Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Maven Companies Inc.

NewBe an early applicant

Sr Data Engineer – Data Ingestion Framework (python.DBT)

Posted

Description

We are looking for an experienced Lead Data Engineer with strong expertise in Python and DBT to design and develop a scalable, reusable, and technology-agnostic data ingestion framework.

The role involves replacing existing ETL capabilities with an open-source technology stack leveraging Python, DBT, and orchestration technologies such as Dagster or Control-M. The solution will utilize AWS S3 as the data Lakehouse storage layer and Snowflake as the target data platform.


The ideal candidate should have hands-on experience building enterprise-grade data ingestion frameworks, developing reusable connectors, implementing metadata-driven pipelines, and integrating heterogeneous source systems with modern cloud data platforms.


Key Responsibilities

1. Data Ingestion Framework Development

· Develop a reusable, scalable, and technology-agnostic data ingestion framework using Python.

· Design modular and extensible components to support multiple source systems.

· Develop metadata-driven and configuration-based ingestion capabilities.

· Build reusable connectors and adapters for different source technologies.

· Implement full and incremental data ingestion patterns, including CDC where applicable.

· Implement robust error handling, logging, auditing, retry, and reprocessing mechanisms.

2. Data Integration & Lakehouse Architecture

· Develop ingestion pipelines to integrate data from heterogeneous source systems into AWS S3 and Snowflake.

· Implement data ingestion from SQL Server, Oracle, PostgreSQL, REST APIs, SFTP, AWS S3, and other source systems.

· Design and implement data lakehouse ingestion patterns using AWS S3.

· Work with file formats such as CSV, JSON, Parquet, and other structured/semi-structured formats.

· Implement schema handling, data validation, partitioning, and data movement strategies.

3. Python & dbt Engineering

· Develop production-grade Python applications and reusable ingestion libraries.

· Build and maintain dbt models for data transformation and integration.

· Implement dbt testing, documentation, incremental models, and reusable macros.

· Optimize Python-based ingestion processes and dbt transformations for performance and scalability.

4. Workflow Orchestration


Required Technical Skills

· Python: Expert-level hands-on experience in Python development, including object-oriented programming, reusable libraries, APIs, error handling, and performance optimization.

· DBT: Strong hands-on expertise in DBT Core, including models, macros, tests, incremental processing, and deployment practices.

· Data Engineering: Extensive experience building enterprise-scale ETL/ELT pipelines and reusable data ingestion frameworks.

· Snowflake: Strong experience with Snowflake architecture, data loading, stages, Snowpipe, SQL, and performance optimization.

· AWS: Hands-on experience with AWS S3, IAM, and cloud-based data integration patterns.

· Data Integration: Experience integrating relational databases, REST APIs, SFTP, and cloud-based storage systems.

· SQL: Advanced SQL development and query optimization skills.

· CI/CD: Experience with Git-based version control, automated builds, testing, and deployment pipelines.


Qualifications

· 10–12 years of overall experience in data engineering and data integration.

· Strong architectural and hands-on engineering capabilities.

· Experience working on enterprise data modernization initiatives.

· Excellent communication and stakeholder management skills.

· Ability to work independently in a distributed onshore/offshore delivery model.


Mandatory Skills

1. Expert-level Python development.

2. Strong hands-on dbt experience.

3. Enterprise data ingestion framework development.

4. Snowflake and AWS S3 experience.

5. Strong SQL and data integration experience.

Preferred: Dagster, Control-M, Apache Iceberg, and SnapLogic migration

Skills

What Senior Data Engineering jobs ask for — and how much of it you have →
Apply

See also

Data Engineering jobs by country — openings, pay and top skills →