Data Engineer w/DevOps
Posted Updated
Description
Techwarely LLC is seeking a semi senior Data Engineer w/DevOps for a Databricks-focused team. You will migrate legacy platforms to Delta Lake and run production pipelines using Python, SQL and PySpark on Databricks, while owning CI/CD, IaC and platform operations.
The role is fully remote and requires collaboration with engineers and stakeholders. You will design and optimize pipelines, manage Unity Catalog, and implement data quality checks to ensure reliability and cost efficiency.
Design and maintain batch and streaming pipelines on Databricks. Migrate legacy data to Delta Lake bronze/silver/gold layers. Develop and optimize Spark jobs, PySpark code and SQL. Implement data quality checks and validation. Build and maintain CI/CD pipelines for Databricks code. Manage Terraform-based infrastructure and Unity Catalog access controls. Set up monitoring and alerts for pipeline reliability. Document pipelines and deployment processes; participate in Scrum with Jira. Experience designing and operating production data pipelines on Databricks. Proven ability to migrate to Delta Lake medallion models. Hands-on CI/CD with Jenkins, GitHub Actions, or Azure DevOps.Skills
Sign in to read and join the discussion.