Senior Data Engineer

We are seeking a highly skilled Senior Data Engineer to join our dynamic team in Karnataka or Andhra Pradesh. The ideal candidate will have extensive experience in data engineering, particularly with Azure Databricks and big data technologies. This role involves designing and implementing data pipelines, optimizing data workflows, and leveraging advanced analytics to drive business insights. Key Responsibilities: - Develop and maintain scalable data pipelines using Azure Databricks, PySpark, and Spark. - Collaborate with data scientists and analysts to understand data requirements and deliver high-quality data solutions. - Implement and manage data storage solutions, including vector databases and embeddings, to support machine learning and AI initiatives. - Utilize Azure OpenAI and Generative AI technologies to enhance data processing and retrieval capabilities. - Ensure data quality and integrity through rigorous testing and validation processes. - Stay updated with the latest trends in data engineering and AI technologies to continuously improve our data infrastructure. Required Skills: - Proficiency in Azure Databricks, PySpark, and Spark. - Strong programming skills in Python and Scala. - Experience with Azure OpenAI, LLM, and Generative AI frameworks. - Knowledge of Retrieval Augmented Generation (RAG) and vector databases. - Excellent problem-solving skills and the ability to work collaboratively in a team environment. If you are passionate about data engineering and eager to work with cutting-edge technologies, we encourage you to apply for this exciting opportunity.