Lead Data Engineer
Posted Updated
Description
Responsibilities:
- Lead and contribute to the development of a self-service data platform for reporting and analytics.
- Design and build data solutions using Databricks, SQL, Python, Spark, and Delta Lake within the Azure ecosystem (Blob Storage, Data Factory, Event Hubs).
- Implement best practices for ETL/ELT processes, focusing on data quality management, partitioning, maintainability, and reusability.
- Collaborate with engineers, product managers, and business leaders to ensure seamless integration of the data platform with systems such as Tableau, Power BI, APIs, and other applications.
- Establish CI/CD pipelines and use infrastructure-as-code tools for deployment and monitoring (Git, Terraform, Azure DevOps/GitHub Actions/Jenkins, Azure Monitor/Datadog).
- Ensure adherence to organizational compliance standards and the Code of Conduct as defined in LogixGarden.
Requirements:
- 8+ years of data engineering experience.
- 3+ years in a senior or principal engineer role, with proven experience leading projects.
- Strong hands-on experience with Databricks and related integrations (Delta Live Tables, Auto Loader, Databricks Workflows / Apache Airflow, Unity Catalogue).
- Expertise in PySpark and/or Scala.
- Strong DataFrame programming experience (Spark, Pandas, NumPy).
- Proven ability to design scalable, secure, and maintainable data pipelines.
- Excellent communication and leadership skills, with a passion for mentoring.
- Solid understanding of data security and governance frameworks, with exposure to HIPAA or SOC 2 compliance preferred.
Preferred Skills:
- AWS cloud experience.
- AWS Databricks implementation expertise.
- Experience designing and implementing data security and governance frameworks.
Skills
- Airflow
- Analytics
- API
- AWS
- Azure
- Azure Data Factory
- Azure DevOps
- Azure Monitor
- CI/CD
- Cloud
- Data Engineering
- Data Pipelines
- Data Quality
- Databricks
- Datadog
- Delta Lake
- DevOps
- ELT
- ETL
- Git
- GitHub
- GitHub Actions
- Hipaa
- Infrastructure as Code
- Jenkins
- NumPy
- pandas
- Power BI
- PySpark
- Python
- Scala
- SOC 2
- Spark
- SQL
- Tableau
- Terraform
- Unity
Sign in to read and join the discussion.