Data Engineer
Posted
Summary
Data Engineer building and maintaining data pipelines on the Databricks platform (Bronze/Silver/Gold layers) using SQL, Python, PySpark, and Spark. Day to day involves data profiling, reconciliation, validation of partner-delivered solutions, data quality/Unity Catalog support, and performance tuning.
Data Engineer – Databricks
Role Overview
The Client Data Engineer will support the design, development, validation, and ongoing ownership of data solutions within the Databricks platform. The role will work closely with internal teams, business SMEs, architects, governance teams, and implementation partners.
Key Responsibilities
- Develop and support Databricks data pipelines using SQL, Python, PySpark, and Spark.
- Support Bronze, Silver, and Gold data layer development.
- Understand source-system data, business rules, mappings, and integration requirements.
- Review and validate solutions delivered by implementation partners.
- Perform data profiling, reconciliation, testing, and troubleshooting.
- Support data quality, metadata, lineage, and Unity Catalog activities.
- Optimize pipelines and queries for performance and scalability.
- Participate in code reviews, CI/CD, deployment, and production support.
- Maintain technical documentation and support knowledge transfer to internal teams.
- Collaborate with BI and analytics teams on downstream data requirements.
Required Skills
- 3+ years of data engineering or ETL/ELT experience.
- Strong SQL and Python skills.
- Experience with Spark or PySpark.
- Experience with cloud-based data platforms.
- Strong understanding of data integration, transformation, and data quality.
- Strong analytical, troubleshooting, and communication skills.
Preferred Skills
- Databricks, Delta Lake, and Unity Catalog.
- Medallion/Lakehouse architecture.
- Azure, AWS, or GCP.
- Git, CI/CD, and DevOps practices.
- Power BI or Tableau experience.
- Databricks certification is a plus.