ADB + ADF + Pyspark + SQL

ADB + ADF + Pyspark + SQL

Core Skills Required

Strong handson experience with Databricks and Apache PySpark

Programming skills in Python PySpark and SQL for efficient data transformation and processing ETL

Solid understanding of ETLELT patterns and data pipeline orchestration

Big Data Processing Experience with big data frameworks specifically Apache Spark for processing large datasets

Data Warehousing Modelling Understanding of data warehousing principles and scalable data modelling techniques

Experience with data engineering best practices and optimization techniques

Handson exposure to enterprise data engineering workflows

Experience supporting production data pipelines at scale

Familiarity with analytics reporting or AIdriven data consumption use cases

Good to have

Experience in working with Opensource storage formats such as Apache Delta Apache Parquet would be beneficial

Awareness of compliance regulations relevant to data handling and processing

Knowledge of data security practices including encryption access control and data governance

Knowledge of other Azure services that integrate with ADF and ADB such as Azure Functions Azure Logic Apps and Azure Event Hubs

Skills

Mandatory Skills : Azure Data Factory, Azure SQL
Good to Have Skills : Databricks

Role & responsibilities

Preferred candidate profile

Perks and benefits