Senior Data Engineer - Azure Data Factory & Databricks
Description
Senior Data Engineer - Azure Data Factory, Microsoft Fabric & Databricks.
Datavail is hiring a Senior Data Engineer to join our high-performing team building and supporting modern data platforms across a wide range of clients and industries.
We are looking for a senior-level data engineer who operates as a trusted technical advisor, someone who goes beyond building pipelines to turn business requirements into sound technical designs, lead platform migrations, and set engineering standards for the team.
The ideal candidate will bring hands-on depth across Azure Data Factory, Microsoft Fabric and Databricks, mentor junior engineers, and own production support for client workloads with a blend of engineering rigor and consultative guidance.
This is a full-time position with full medical benefits.
Azure Data Factory, Microsoft Fabric & Databricks Expertise:
- Metadata-driven ADF pipelines: designing reusable, parameterized pipelines for batch and incremental loads.
- Hybrid ingestion & integration runtimes: ingesting from on-prem databases, files, REST APIs, SaaS applications and cloud storage.
- ADF orchestration & monitoring: schedule, tumbling window and event-based triggers; orchestrating Databricks notebooks and jobs.
- Fabric Lakehouse & Warehouse design: medallion (Bronze/Silver/Gold) architecture on OneLake; Warehouse objects in T-SQL.
- Fabric data engineering: Data Pipelines and Dataflows Gen2 for ingestion and transformation; PySpark and Spark SQL.
- Semantic modeling & capacity management: star-schema and Direct Lake semantic models for high-performance Power BI reporting.
- Databricks development: scalable processing in PySpark, Spark SQL and Python; Databricks Workflows/Jobs.
- Delta Lake design & Spark performance tuning: MERGE/upsert, SCD Type 1 and Type 2, schema evolution and Time Travel.
- Unity Catalog & platform configuration: catalogs, schemas, access controls, lineage and external locations.
- CI/CD across platforms: ADF source control with Git; Fabric Git integration; Databricks Asset Bundles.
In this role, you will:
- Serve as a trusted technical advisor to clients, turning business requirements into technical designs.
- Lead migrations from legacy ETL (SSIS, Informatica, on-prem SQL) and from Azure Synapse to Microsoft Fabric and Databricks.
- Design, develop and implement scalable data pipelines using Microsoft Fabric Data Factory, Azure Data Factory and Azure Databricks.
- Set up security across all platforms using Key Vault, managed identities, RBAC, and row and column-level security.
- Build ETL/ELT workflows to ingest data from databases, ERP and CRM systems.
- Develop and maintain enterprise data warehouses and subject-area data marts.
- Provide production support for Azure, Fabric and Databricks workloads.
- Collaborate directly with client technical teams and stakeholders.
- Participate in the on-call rotation to support critical production issues.
- Design and implement data quality, validation and reconciliation checks.
- Proactively monitor pipeline performance and platform costs.
- Build reusable frameworks and templates for ingestion, logging and auditing.
- Run discovery and onboarding for new client environments.
- Lead data platform modernization roadmaps.
- Partner with BI developers and analysts to deliver reliable, well-modeled data for Power BI reporting.
- Coordinate with offshore and onshore team members.
Required Qualifications:
- 12+ years of overall experience, including 6+ years of hands-on experience with Azure Data Factory and Databricks, and 2+ years of experience delivering production workloads using Microsoft Fabric.
- Expert-level PySpark and Spark SQL writing and optimizing large-scale transformations.
- Advanced SQL/T-SQL complex joins, window functions, CTEs, MERGE logic, and query tuning.
- Strong Python for data engineering modular, testable code, reusable libraries, and unit testing with pytest.
- Built and supported metadata/config-driven ADF frameworks in production.
- Hands-on Delta Lake implementation, MERGE/upsert, SCD Type 1 and Type 2.
- Production experience with Auto Loader, Structured Streaming or Delta Live Tables.
- Hands-on Unity Catalog administration.
- Built Fabric Lakehouses/Warehouses end to end.
- Dimensional data modeling like star and snowflake schemas.
- Azure platform skills like ADLS Gen2, Azure SQL, Key Vault, managed identities.
- CI/CD implementation - Git branching strategies, Azure DevOps YAML pipelines or GitHub Actions.
- Infrastructure as Code with Terraform or Bicep.
- Data quality and observability implementation.
- Proven track record migrating SSIS, Informatica or Synapse workloads.
- Hands-on production support experience.
- Clear verbal and written communication skills.
Good to Have:
- Snowflake, SSIS and SSRS, Streaming technologies (Fabric Real-Time Intelligence, Event Hubs, Kafka), Power BI and DAX.
Certifications:
- Preferred: Microsoft Certified: Fabric Data Engineer Associate (DP-700), Fabric Analytics Engineer Associate (DP-600), Azure Data Engineer Associate (DP-203), Databricks Certified Data Engineer Professional, or SnowPro Core.
Skills
- Analytics
- API
- Azure
- Azure Data Factory
- Azure DevOps
- Azure Synapse
- Bicep
- CI/CD
- Cloud
- CRM
- Data Engineering
- Data Modeling
- Data Pipelines
- Data Quality
- Databricks
- Delta Lake
- DevOps
- ELT
- ERP
- ETL
- Git
- GitHub
- GitHub Actions
- Infrastructure as Code
- Kafka
- Lakehouse
- Microsoft Fabric
- Observability
- Power BI
- PySpark
- pytest
- Python
- RBAC
- REST
- SaaS
- Snowflake
- Spark
- SQL
- SQL Server
- SSIS
- SSRS
- Terraform
- Unit Testing
- Unity
- Unity Catalog
- Vault
- YAML
Sign in to read and join the discussion.