Lead Data Engineer (Banking)
Posted Updated
Summary
Lead Data Engineer for a large financial institution (hybrid in Miami): owns delivery of 4-6 concurrent data pipeline projects, directs an 8-12 person consultant team, and leads migration from on-prem NiFi/Airflow/dbt/SQL pipelines to AWS and Databricks in a regulated banking environment.
ITTConnect is seeking a Lead Data Engineer for a direct-hire full time position with a client that is a large financial institution.
Position is hybrid in Miami.
The Lead Data Engineer turns business and domain needs into secure, observable, reusable, and documented pipelines. The role combines hands-on technical leadership, portfolio delivery, consultant coordination, engineering standards, and stakeholder engagement in a regulated bank.
Key Responsibilities:
Portfolio & Delivery Leadership
- Own integrated delivery for 4-6 concurrent projects, balancing priorities, dependencies, capacity, risks, and production commitments, working under a scaled agile framework.
- Turn transformation needs into technical scope, milestones, acceptance criteria, and measurable outcomes.
- Provide portfolio visibility through dashboards, risk logs, dependency tracking, and escalation.
- Ensure releases meet change, security, testing, documentation, and production-readiness requirements.
Team & Consultant Coordination
- Direct an 8-12 person consultant team through work allocation, onboarding, delivery follow-up, and quality review.
- Set an accountable rhythm with backlog refinement, design and code reviews, stand-ups, demos, and retrospectives.
- Build capability through reusable patterns, playbooks, pairing, coaching, and knowledge transfer.
- Partner with vendor and bank leaders to manage capacity, capability, performance, continuity, and key-person risks.
On-Premises Platform Execution
- Guide solutions using NiFi for ingestion, Airflow for orchestration, dbt for transformation and testing, and SQL for processing and validation.
- Improve pipeline stability through modular design, automated tests, version control, disciplined deployment, observability, and incident follow-up.
- Prioritize technical debt by business criticality, operational risk, migration value, and delivery impact.
Cloud Transition & Re-Engineering
- Follow the AWS and Databricks operating model in line with architecture, security, governance, networking, and compliance standards.
- Lead cloud-native pipeline delivery and phased re-engineering of on-premises pipelines.
- Redesign and remodeling instead of lifting and shifting when it improves scale, reliability, cost transparency, reuse, lineage, or supportability.
- Define migration waves, readiness, parallel runs, reconciliation, cutover, rollback, and decommissioning.
Stakeholder & Domain Partnership
- Align delivery with business and data owners, stewards, architects, security, infrastructure, risk, compliance, and operations.
- Lead cross-domain requirements and design discussions, resolving ambiguity and clarifying trade-offs.
- Explain decisions, delivery health, risks, and business impact to technical and executive audiences.
Requirements
- 15+ years of experience in IT.
- 8+ years in data engineering, platform delivery, analytics engineering, or a related field
- 3+ years in technical or delivery leadership.
- Experience coordinating parallel projects and distributed or consulting teams in complex enterprises.
- Strong SQL expertise and practical experience with NiFi, Airflow, dbt, or comparable tools.
- Experience building production pipelines with automated tests, quality controls, monitoring, lineage, documentation, and support.
- Working knowledge of AWS and Databricks architecture, lakehouse patterns, security, deployment, and migration.
- Experience in regulated financial services or another control-intensive environment.
- Strong English communication, planning, problem-solving, and stakeholder management;
- Fluency in Portuguese or Spanish is a plus.
Preferred Qualifications
- Experience with Unity Catalog, Delta Lake, Databricks Workflows, CI/CD, infrastructure as code, and cloud observability.
- Knowledge of AWS data-platform services, including S3, IAM, KMS, CloudWatch, and private networking.
- Experience with catalogs, metadata, lineage, data quality, and DataOps or DataSecOps.
- Familiarity with dimensional modeling, data vault, medallion/lakehouse, streaming, and data products.
- Relevant AWS, Databricks, data engineering, or agile delivery certifications.
Skills
- Acceptance Criteria
- Agile
- Airflow
- Analytics
- AWS
- CI/CD
- Cloud
- Cloud Native
- CloudWatch
- Data Engineering
- Data Quality
- Databricks
- DataOps
- dbt
- Delta Lake
- Dimensional Modeling
- IAM
- Infrastructure as Code
- Lakehouse
- Networking
- NiFi
- Observability
- S3
- SAFe Agile
- SQL
- Stakeholder Management
- Unity
- Unity Catalog
- Vault
- Version Control