Azure Databricks Architect

Role & responsibilities

We are seeking an experienced and detail-oriented Azure Databricks Data Engineer to join our growing data team. In this role, you will be responsible for designing, developing, implementing, and maintaining high-performance, scalable data pipelines and solutions within the Microsoft Azure ecosystem. You will collaborate closely with cross-functional teams to translate business requirements into efficient ETL/ELT workflows, ensuring optimal data quality, performance, and cost-efficiency.

Key Responsibilities

• Data Integration & Pipelines: Design, build, and deploy end-to-end scalable data pipelines and robust ETL/ELT workflows using Azure Databricks, PySpark, Spark SQL, and Azure Data Factory (ADF).
• Architecture Implementation: Build and optimize lakehouse architectures leveraging Delta Lake (Medallion architecture: Bronze, Silver, Gold layers) to manage structured, semi-structured (JSON, XML), and unstructured data.
• Performance Tuning: Write highly efficient PySpark and SQL code. Perform partition tuning, caching optimizations, and cluster scaling to minimize execution runtimes and operational cloud costs.
• Data Governance & Security: Set up, configure, and govern data workspaces using Unity Catalog. Secure sensitive data using Azure Key Vault-backed secrets and implement role-based access controls (RBAC).
• CI/CD & DevOps: Implement modern DevOps practices, including version control with Git, and automate deployments through CI/CD pipelines (e.g., Azure DevOps or GitHub Actions).
• Monitoring & Troubleshooting: Set up proactive monitoring, logging, and alerting systems via Azure Monitor or Databricks Workflows to diagnose and resolve complex data pipeline failures.

Required Technical Skills & Qualifications

• Education: Bachelors degree in Computer Science, Data Engineering, Information Technology, or a related discipline.
• Big Data & Cloud: 5+ years of hands-on data engineering experience, with at least 3 years explicitly focused on Azure Databricks, Apache Spark, and Azure Data Lake Storage (ADLS Gen2).
• Programming Languages: Proficiency in Python / PySpark and advanced SQL for complex transformations, data modeling, and performance tuning.
• Data Tools: Direct experience working with Delta Lake and Azure Synapse Analytics.
• Methods: Solid understanding of dimensional data modeling, data warehousing principles, and data quality validation frameworks.
• Agile Ecosystem: Experience working in an Agile/Scrum environment with cross-functional data scientists, analysts, and business stakeholders

Contact-Soniya

soniya05.mississippiconsultants@gmail.com