Data Engineer (Databricks)
Summary
Data Engineer designing, building, and maintaining scalable ETL pipelines on Databricks, Delta Lake, and the Azure cloud to support enterprise data, reporting, and analytics. Day-to-day work spans ingestion, transformation, monitoring, and deployment in collaboration with analytics, product, and infrastructure teams.
We are looking for an experienced Data Engineer to design, develop, and maintain scalable data pipelines on Databricks and cloud platforms. This role will work closely with analytics, product, and infrastructure teams to support enterprise data, reporting, and analytics initiatives.
Key Responsibilities
- Develop and maintain ETL pipelines for centralized data platforms such as Delta Lake.
- Build ingestion, transformation, validation, and loading processes.
- Integrate data from databases, APIs, streaming platforms, log files, and external providers.
- Design and optimize scalable data processing solutions.
- Implement data governance, monitoring, and reliability standards.
- Develop monitoring tools, alerts, and automated error-handling mechanisms.
- Support SIT, UAT, data profiling, validation, and production deployment.
- Collaborate with business, analytics, product, and infrastructure stakeholders to deliver data solutions.
Required Skills
✅ 3+ years of Data Engineering experience
✅ Strong experience with:
- Databricks
- PySpark
- Spark SQL
- Databricks Notebooks & Jobs
- Data Modelling
- Distributed Computing Architectures
- SQL Performance Optimization
✅ Experience with:
- Azure Data Platform
- Azure Data Factory (ADF), Airflow, or similar orchestration tools
- Batch and Real-Time Data Processing
- Git, CI/CD, and DevOps practices
- Scrum/Agile methodologies
✅ Knowledge of streaming technologies such as:
- Apache Kafka
- Apache Flink
- AWS Kinesis
Preferred Certifications
- Databricks Certified Data Engineer Associate
- Databricks Certified Data Engineer Professional
Why Join
- Opportunity to work on enterprise-scale cloud and data transformation projects.
- Exposure to modern Azure and Databricks technologies.
- Collaborative environment focused on data innovation and engineering excellence.