Senior Data Engineer
Posted Updated 2
views
Summary
The Senior Data Engineer will design and build scalable data platforms, ETL/ELT pipelines, and data lakes using PySpark and Databricks within Azure and AWS cloud environments. The role involves implementing real-time data ingestion frameworks and ensuring data quality, security, and governance.
Job Summary
We are looking for a Senior Data Engineer with strong experience designing and delivering scalable data platforms across Microsoft Azure and AWS ecosystems. The candidate will be responsible for building high-performance data pipelines, data lakes, and real-time data ingestion frameworks using PySpark and Databricks.
Key Responsibilities
- Design and develop scalable data platforms on Azure and AWS.
- Build and optimize high-performance ETL/ELT pipelines using PySpark and Databricks.
- Develop and manage cloud-based data lakes and data warehouses.
- Design real-time and batch data ingestion frameworks.
- Implement data processing solutions using Apache Spark/PySpark.
- Optimize data pipelines for performance, scalability, and reliability.
- Work with cloud services such as Azure Data Factory, ADLS, Azure Synapse, AWS S3, Glue, and EMR.
- Implement data quality, security, monitoring, and governance standards.
- Collaborate with Data Architects, Engineers, and business stakeholders.
Required Skills
- 5+ years of experience in Data Engineering.
- Strong hands-on experience with Azure and/or AWS data platforms.
- Excellent PySpark and Databricks experience.
- Strong knowledge of Data Lakes, ETL/ELT, and real-time data ingestion.
- Experience with SQL and Python.
- Knowledge of Apache Spark and distributed data processing.
- Experience with cloud data services and data architecture.
- Good understanding of CI/CD, Git, and Agile methodologies.
Preferred Skills
- Azure Data Factory / Azure Synapse / ADLS
- AWS S3 / Glue / EMR
- Kafka or other real-time streaming technologies
- Delta Lake
- Terraform / Infrastructure as Code
- Data governance and security