Senior Data Engineer (Databricks)
Experience: 8+ yrs
We are looking for a Senior Data Engineer with strong expertise in Databricks and modern data platforms. The ideal candidate will be responsible for building scalable data pipelines, optimizing data workflows, and enabling data-driven decision-making across the organization.
- Design, develop, and maintain scalable data pipelines using Databricks and Python
- Build and optimize ETL workflows for structured and unstructured data
- Work with Databricks Lakehouse architecture and implement best practices
- Manage and implement data governance using Unity Catalog
- Integrate AWS data services such as S3, IAM, VPC, and other relevant services
- Collaborate with cross-functional teams to deliver high-quality data solutions
- Build dashboards and support data visualization requirements
- Ensure data quality, reliability, and performance optimization
- Strong experience with Databricks platform
- Proficiency in Python for data engineering and ETL pipelines
- Hands-on experience with Unity Catalog
- Good knowledge of AWS services (S3, IAM, VPC, Glue/Lambda – preferred)
- Solid understanding of Data Lake / Lakehouse architecture
- Experience in building dashboards and reporting solutions
- Experience with REST API development (Flask, FastAPI, etc.)
- Knowledge of authentication/authorization (OAuth, API keys, IAM roles)
- Strong query optimization and performance tuning skills
- Experience with PySpark optimization
- Exposure to ML/AI pipelines
- Familiarity with Databricks AI/BI capabilities
- Strong problem-solving and analytical skills
- Excellent communication and collaboration abilities