Data Engineer - Azure Databricks/Apache Spark
Summary
Data Engineer in Jaipur who designs, builds, and optimizes large-scale data pipelines using Databricks (Apache Spark, Delta Lake) on cloud platforms like Azure, AWS, or GCP. Day-to-day includes enforcing data quality and governance (e.g., Unity Catalog, Great Expectations) and monitoring/debugging pipeline jobs.
Key Responsibilities :
- Design, develop, and manage data pipelines using Databricks (Spark, Delta Lake).
- Optimize large-scale data processing workflows for performance and reliability.
- Collaborate with Data Scientists, Analysts, and Business Stakeholders to gather requirements and deliver actionable insights.
- Maintain and enforce data quality and integrity across multiple data sources and systems.
- Work with cloud data platforms such as Azure, AWS, or GCP.
- Implement data governance and lineage tracking using tools like Unity Catalog, Great Expectations, or similar.
- Monitor, debug, and troubleshoot data pipelines and jobs.
Required Qualifications :
- 3+ years of professional experience in data engineering or similar roles.
- Strong experience with Databricks, Apache Spark, and Delta Lake.
- Proficient in SQL, Python, and distributed data processing concepts.
- Experience working with cloud platforms (Azure/AWS/GCP) and cloud-native tools.
- Hands-on experience with ETL/ELT processes, data warehousing, and modern data stack.
- Familiarity with CI/CD practices and version control tools (e.g., Git).
- Strong problem-solving skills and ability to work independently or in a team environment.