Data Engineer - AI/Automation, PySpark & Cloud ETL
Summary
Data engineer at a data solutions firm building reusable PySpark frameworks and scalable ETL/ELT pipelines on Databricks across Azure and AWS. Day to day: distributed data processing with Python/PySpark, batch and near real-time ingestion from healthcare systems, and secure data integration plus backend data layer design.
Halo Labs is a data solutions firm seeking a data engineer to design, develop, and maintain reusable PySpark frameworks and scalable ETL/ELT pipelines across Azure and AWS. You will build distributed data processing with Python and PySpark on Databricks, and develop batch and near real-time ingestion pipelines from healthcare systems and external platforms.
You will design secure data integration patterns, collaborate with software engineers, and contribute to backend data layer design, with