Senior Data Engineer
Posted
Summary
Senior Data Engineer designing and delivering scalable data platforms on Azure and AWS. Responsibilities include building high-performance ETL/ELT pipelines, data lakes, and real-time ingestion frameworks using PySpark and Databricks.
Job Summary
We are looking for a Senior Data Engineer with strong experience designing and delivering scalable data platforms across Microsoft Azure and AWS ecosystems. The candidate will be responsible for building high-performance data pipelines, data lakes, and real-time data ingestion frameworks using PySpark and Databricks.
Key Responsibilities
Design and develop scalable data platforms on Azure and AWS.
Build and optimize high-performance
ETL/ELT pipelines
using PySpark and Databricks.
Develop and manage cloud-based
data lakes and data warehouses .
Design real-time and batch data ingestion frameworks.
Implement data processing solutions using
Apache Spark/PySpark .
Optimize data pipelines for performance, scalability, and reliability.
Work with cloud services such as
Azure Data Factory, ADLS, Azure Synapse, AWS S3, Glue, and EMR .
Implement data quality, security, monitoring, and governance standards.
Collaborate with Data Architects, Engineers, and business stakeholders.
Required Skills
5+ years of experience in Data Engineering.
Strong hands-on experience with
Azure and/or AWS
data platforms.
Excellent
PySpark and Databricks
experience.
Strong knowledge of
Data Lakes, ETL/ELT, and real-time data ingestion .
Experience with
SQL and Python .
Knowledge of Apache Spark and distributed data processing.
Experience with cloud data services and data architecture.
Good understanding of CI/CD, Git, and Agile methodologies.
Preferred Skills
Azure Data Factory / Azure Synapse / ADLS
AWS S3 / Glue / EMR
Kafka or other real-time streaming technologies
Delta Lake
Terraform / Infrastructure as Code
Data governance and security
Key Responsibilities
Design and develop scalable data platforms on Azure and AWS.
Build and optimize high-performance
ETL/ELT pipelines
using PySpark and Databricks.
Develop and manage cloud-based
data lakes and data warehouses .
Design real-time and batch data ingestion frameworks.
Implement data processing solutions using
Apache Spark/PySpark .
Optimize data pipelines for performance, scalability, and reliability.
Work with cloud services such as
Azure Data Factory, ADLS, Azure Synapse, AWS S3, Glue, and EMR .
Implement data quality, security, monitoring, and governance standards.
Collaborate with Data Architects, Engineers, and business stakeholders.
Required Skills
5+ years of experience in Data Engineering.
Strong hands-on experience with
Azure and/or AWS
data platforms.
Excellent
PySpark and Databricks
experience.
Strong knowledge of
Data Lakes, ETL/ELT, and real-time data ingestion .
Experience with
SQL and Python .
Knowledge of Apache Spark and distributed data processing.
Experience with cloud data services and data architecture.
Good understanding of CI/CD, Git, and Agile methodologies.
Preferred Skills
Azure Data Factory / Azure Synapse / ADLS
AWS S3 / Glue / EMR
Kafka or other real-time streaming technologies
Delta Lake
Terraform / Infrastructure as Code
Data governance and security