Sr. Databricks Data Engineer

Halvik Corp delivers a wide range of services to 13 executive agencies and 15 independent agencies. Halvik is a highly successful WOB business with more than 50 prime contracts and 500+ professionals delivering Digital Services, Advanced Analytics, Artificial Intelligence/Machine Learning, Cyber Security and Cutting-Edge Technology across the US Government. Be a part of something special!

Position Overview

The Senior Databricks Data Engineer / Technical Lead will support the U.S. Department of Transportation (DOT) data modernization effort by designing, building, migrating, and operating secure, scalable data pipelines in a Databricks Lakehouse. This is a highly hands-on role focused on legacy and external-source migration, metadata-driven ingestion, Medallion Architecture, data quality and reconciliation, production operations, and transition to the AWS/OneDOT environment.

Core Responsibilities

• Strategic Execution: Lead the design and implementation of reusable ingestion and transformation patterns for legacy and external data sources into Databricks, including support for migration to the AWS/OneDOT environment.
• Operational Oversight: Build and operate Bronze, Silver, and Gold pipelines; manage batch and incremental loads, orchestration, dependencies, retries/recovery, monitoring, alerting, and production support.
• Collaboration: Partner with the Technical Project Manager, Data Architects, business/data SMEs, BI developers, Cloud/DevSecOps, security, and governance teams to translate requirements into implementable data solutions.
• Technical Performance: Develop and optimize Python, PySpark, and SQL workloads using Databricks, Delta Lake, Unity Catalog, Auto Loader and/or Lakeflow capabilities, and Databricks Jobs/Workflows with a focus on reliability, performance, and cost.
• Data Quality & Governance: Implement source-to-target reconciliation, data validation, schema evolution, lineage, access controls, documentation, and reusable engineering standards for production data pipelines.

Minimum Requirements

• Education: Bachelor's degree in computer science, Data Engineering, Information Technology, Engineering, or a related field; equivalent relevant experience may be considered.
• Experience: Minimum of 7 years of professional data engineering experience, including significant hands-on experience with Databricks/Spark and production data pipelines.
• Technical Proficiency: Advanced Python, PySpark, and SQL skills; strong experience with Databricks, Delta Lake, AWS/S3, batch/file ingestion (CSV/JSON), REST APIs, Git/CI/CD, and data pipeline orchestration.
• Compliance: Must meet applicable DOT contract security and suitability requirements and demonstrate secure handling of PII and other sensitive data, including access controls and secrets management.

• Certifications: Relevant Databricks and/or AWS data engineering certification preferred; equivalent demonstrated hands-on experience will be considered.

Preferred Expertise

• Demonstrated success modernizing legacy data platforms (Sybase/SAP IQ or similar) into Databricks, including source-to-target reconciliation, cutover, and production stabilization.
• Experience with Unity Catalog, metadata/configuration-driven ingestion, Auto Loader/Lakeflow, Databricks Jobs/Workflows, lineage/cataloging tools, and scalable multi-source ingestion frameworks.
• Federal/DOT delivery experience; familiarity with Agile/Kanban, production O&M, cloud cost optimization, and data governance. Streaming and AI/ML data pipeline exposure is a plus.