Data Engineer: Databricks Lakehouse & PySpark/Delta - Remote
Summary
A middle-level Data Engineer modernizes a 15-year-old data warehouse into a Databricks Lakehouse, building batch and streaming pipelines with PySpark and Delta Lake using a medallion architecture for analytics and ML. Day to day involves writing clean Python/SQL, optimizing Spark jobs for performance and cost, and using AI assistants (Claude, GitHub Copilot) on a remote team.
AgileEngine is seeking a Middle Data Engineer to modernize a 15-year-old data warehouse into a Databricks Lakehouse. You will build batch and streaming pipelines with PySpark/Delta Lake, following a medallion architecture for analytics and ML use cases.
You will work with Claude and GitHub Copilot to accelerate development, write clean Python and SQL, and optimize Spark jobs for performance and cost. This remote role offers flexible hours and collaboration with global teams.