Senior Data Engineer- SQL Server
Description
Senior Data Engineer- SQL Server to Databricks Migration
Job Type: Contract (6-12 Months)- Possible extension
Location: Philadelphia, PA / Vienna, VA - Hybrid / Open to remote job
Job description:
ü Job summary
ü We are seeking a senior data engineer with deep Microsoft SQL Server experience to assess and convert a legacy warehouse to Azure Databricks.
ü You will read complex T-SQL stored procedures and SSIS packages, document the business rules they contain, rebuild them as Databricks pipelines, and reconcile the results before the legacy platform is retired.
Key responsibilities
ü Assess T-SQL stored procedures, SSIS packages and scheduled jobs for complexity, dependencies, and embedded business rules.
ü Recommend, for each object, whether to retire, convert or redesign it.
ü Convert legacy logic to Databricks SQL and PySpark pipelines using agreed patterns and conversion tooling.
ü Review and correct automatically converted code.
ü Build and run reconciliation between SQL Server and Databricks and resolve differences with business analysts.
ü Migrate job schedules and dependencies to Databricks workflows.
ü Support cutover and the evidence needed to decommission each legacy component.
ü Document data flows, mappings and design decisions in the client's repositories.
Required qualifications
ü 8+ years in data engineering or database development, including 5+ years of hands-on T-SQL and SSIS.
ü 2+ years building pipelines on Databricks, or an equivalent Spark-based lakehouse platform, with Delta Lake.
ü At least one completed migration from SQL Server or another warehouse to a cloud data platform.
ü Strong SQL performance tuning and the ability to reverse-engineer undocumented code.
ü Experience with data reconciliation and testing at scale.
ü Git-based development with pull requests and automated deployment.
Preferred qualifications
ü Databricks Lakeflow pipelines and jobs, and Unity Catalog.
ü Code-conversion tools such as Databricks Lakebridge.
ü Databricks Data Engineer Associate or Professional certification.
ü Financial services or wealth management data.
ü Experience with enterprise schedulers such as SQL Agent or ActiveBatch.
Skills
Sign in to read and join the discussion.