Data Engineer
Description
The Opportunity
We are looking for an experienced Data Engineer to design, build, and operate scalable data solutions within a high-volume environment powered by Microsoft Azure and Microsoft Fabric.
You will collaborate with architects, analysts, data scientists, reporting teams, and business stakeholders to develop reliable and production-ready capabilities across data ingestion, transformation, storage, integration, analytics, and real-time processing.
This is a hands-on engineering role requiring strong expertise across the modern Microsoft data ecosystem, with a particular focus on Microsoft Fabric, Azure data services, Python/PySpark, SQL, Lakehouse architecture, and scalable data pipelines.
What You’ll Do
- Design, build, test, deploy, and maintain scalable batch and real-time data pipelines.
- Develop ingestion and transformation workflows using Azure Data Factory, Microsoft Fabric Data Factory, notebooks, Azure Databricks, Python, PySpark, and SQL.
- Build and maintain Lakehouse, data lake, and data warehouse solutions using OneLake, Azure Data Lake Storage, Fabric Lakehouse, Fabric Warehouse, and Azure Synapse Analytics.
- Develop reusable data models, transformation frameworks, APIs, and integration components.
- Optimize pipelines, Spark workloads, queries, storage formats, and compute resources for performance, reliability, and cost.
- Implement data quality checks, reconciliation controls, monitoring, alerting, logging, lineage, and robust error-handling mechanisms.
- Support streaming and event-driven architectures using Event Hubs, Service Bus, Fabric Real-Time Intelligence, or comparable technologies.
- Apply security, privacy, access-control, governance, and data-retention requirements across the data platform.
- Build and maintain automated CI/CD pipelines using Azure DevOps, Git, and infrastructure deployment practices.
- Collaborate with technical and business stakeholders to translate requirements into scalable, production-ready data solutions.
- Troubleshoot production incidents, perform root-cause analysis, and continuously improve platform stability and supportability.
- Create and maintain technical documentation, data mappings, operational procedures, and solution diagrams.
What You Bring to the Table
- Bachelor’s degree in Computer Science, Software Engineering, Data Engineering, Information Technology, or a related field.
- 5+ years of experience in data engineering, data integration, or cloud data-platform development.
- Strong hands-on experience designing and implementing high-volume, scalable data pipelines in Microsoft Azure.
- Hands-on experience with Microsoft Fabric, including OneLake, Lakehouse, Data Factory pipelines, notebooks, Spark, Data Engineering, Data Warehouse, Real-Time Intelligence, semantic models, and Power BI integration.
- Advanced programming skills in Python or PySpark, combined with strong SQL proficiency.
- Experience with Azure Data Factory, Azure Databricks, Azure Data Lake Storage, Azure Synapse Analytics, and Azure SQL.
- Strong understanding of ETL/ELT patterns, dimensional modelling, data warehousing, Lakehouse architecture, Delta Lake, APIs, and batch and streaming processing.
- Experience with Git, Azure DevOps, automated testing, CI/CD, and release-management practices.
- Understanding of data security, governance, lineage, observability, and platform monitoring.
- Strong analytical, troubleshooting, collaboration, and technical communication skills.
Core Skills Required
- Microsoft Fabric
- Microsoft Azure
- OneLake and Lakehouse architecture
- Azure Data Factory / Fabric Data Factory
- Azure Databricks
- Azure Data Lake Storage
- Azure Synapse Analytics
- Fabric Data Warehouse
- Python / PySpark
- SQL and Spark
- Delta Lake
- ETL / ELT
- Batch and streaming data processing
- Git and Azure DevOps
- CI/CD and automated deployment
- Data governance, lineage, security, and observability
Preferred experience includes:
- Cloud data-platform modernization, migration, or transformation programs.
- Medallion architecture, data mesh, metadata-driven pipelines, and reusable engineering frameworks.
- Infrastructure as code with Bicep, Terraform, or ARM templates.
- Containerization, orchestration, APIs, DevOps, or DataOps practices.
- Regulated, enterprise-scale, or mission-critical environments.
- Relevant Microsoft certifications, such as Microsoft Certified: Fabric Data Engineer Associate or Azure Data Engineer Associate.
Skills
- Analytics
- API
- ARM Templates
- Azure
- Azure Data Factory
- Azure DevOps
- Azure Synapse
- Bicep
- CI/CD
- Cloud
- Containerization
- Data Engineering
- Data Governance
- Data Ingestion
- Data Lake
- Data Mesh
- Data Pipelines
- Data Quality
- Data Warehousing
- Databricks
- DataOps
- Delta Lake
- DevOps
- Dimensional Modeling
- ELT
- ETL
- Event Driven Architecture
- Git
- Infrastructure as Code
- Lakehouse
- Medallion Architecture
- Microsoft Fabric
- Observability
- Power BI
- PySpark
- Python
- Root Cause Analysis
- Spark
- SQL
- Terraform
- Test Automation
Sign in to read and join the discussion.