Data Engineer
Posted 14
views
Summary
Data Engineer building and optimizing Azure data pipelines (ADF, Databricks, Delta Lake) using Python, PySpark, and SQL for batch and near-real-time processing.
- Build data ingestion and transformation frameworks
- Collaborate with architects analysts BI developers and engineering teams
- Create reusable components and frameworks
- Deploy and manage ADF Databricks and Azure data components across environments
- Design batch and near real time processing solutions
- Design develop test and deploy data pipelines on Azure
- Develop data transformation logic using Python PySpark and SQL
- Implement CI CD for data engineering components with Azure DevOps
- Implement data quality checks error handling logging and monitoring
- Optimize Databricks workloads using Apache Spark and Delta Lake
- Optimize SQL queries Spark jobs and data pipelines
- Own data pipeline projects with minimal supervision
- Perform data analysis, profiling, and validation
- Prepare technical design and operational documentation
- Process structured semi structured and unstructured data
- Support production deployments and troubleshoot post production issues
- Translate business and technical requirements into data engineering solutions
- Troubleshoot pipeline failures performance issues and data discrepancies