Data Engineer
Summary
The Data Engineer will build and maintain batch and streaming data pipelines using Spark and the Azure Databricks platform. Key responsibilities include managing data workflows, orchestration, and utilizing Unity Catalog for governance.
Job Overview: Pay Range: $50.00hr - $55.00hr Requirement/Must Have: 5 years building batch and/or streaming pipelines with Spark. Deep understanding of Spark fundamentals including lazy evaluation, transformations vs. actions, partitions, shuffles, joins, and caching. Experience with Databricks platform on Azure including Unity Catalog (catalogs/schemas/tables, permissions, external locations, storage credentials, lineage). Proficiency with Databricks Jobs/Workflows for task orchestration, clus…