Data Engineer (Databricks)
Summary
The Data Engineer will design and maintain scalable data pipelines and ETL/ELT processes using Databricks, Python, and Azure Data Factory. The role involves collaborating with cross-functional teams to ensure data quality and accessibility for business analytics.
We are hiring a Data Engineer!
Summary
As a Data Engineer, you will design, develop, and maintain scalable data solutions that support data generation, ingestion, transformation, and processing. You will be responsible for building efficient data pipelines, ensuring data quality, and implementing ETL/ELT processes across various systems and platforms. The role requires close collaboration with cross-functional teams to improve data accessibility, reliability, and overall business insights.
Roles & Responsibilities
- Serve as a Subject Matter Expert (SME) for data engineering solutions and platforms.
- Collaborate with team members and stakeholders to deliver high-quality data solutions.
- Take ownership of technical decisions and contribute to key project initiatives.
- Develop, optimize, and maintain scalable data pipelines and workflows.
- Design and implement ETL/ELT processes for structured and unstructured data.
- Monitor, troubleshoot, and resolve data quality and pipeline performance issues.
- Implement best practices for data governance, security, and compliance.
- Create and maintain technical documentation for data processes, architectures, and workflows.
- Work closely with data analysts, data scientists, and business teams to support reporting and analytics requirements.
Professional & Technical Skills
Must-Have Skills
- Minimum of 4 years of experience in Databricks and data engineering technologies.
- Experience working with large-scale data processing and distributed systems is preferred.
- Experience in Databricks (Unity Catalog, Delta Live Tables, Auto Loader)
- Experience in Python / PySpark
- Experience in SQL
- Experience in Azure Data Factory
Good-to-Have Skills
- Experience with cloud-based data platforms and services, preferably Azure.
- Strong understanding of data modeling and database design principles.
- Knowledge of data warehousing concepts and best practices.
- Experience with performance tuning and optimization of big data workloads.
- Familiarity with CI/CD processes and DevOps practices in data engineering.