Senior Data Engineer — Microsoft Fabric & Data Pipelines
Summary
Data engineer (placed via recruiter Praesignis for LexisNexis) joining the AI Platforms team in Randburg, South Africa, to take ownership of the Microsoft Fabric environment: building and maintaining batch and near-real-time ingestion pipelines, transforming data with SQL/Python/PySpark, and supporting Power BI reporting that feeds AI/ML initiatives.
About the Business
LexisNexis Legal & Professional® provides legal, regulatory, and business information and analytics that help customers increase their productivity, improve decision-making, achieve better outcomes, and advance the rule of law around the world. As a digital pioneer, the company was the first to bring legal and business information online with its Lexis® and Nexis® services.
About the Role
Our client is seeking a skilled Data Engineer to join their expanding AI Platforms team in Randburg. This role is integral to building and maintaining robust data infrastructure that powers our advanced AI and machine learning initiatives. You will be responsible for designing, developing, and optimizing data pipelines, ensuring data quality, and making large datasets accessible for analysis and model training. The ideal candidate has a strong background in data engineering principles and a passion for enabling data-driven innovation within a collaborative, fast‑paced technology environment.
Requirements
- Minimum Education (essential)
- Bachelor's degree in Computer Science, Information Systems, Data Engineering or a related field.
- Relevant Microsoft Fabric and/or Azure data certification.
- Minimum applicable experience (years):
- 0-1 years of practical data engineering experience, including strong recent hands‑on experience with Microsoft Fabric.
- Required nature of experience:
- Strong hands‑on experience with Microsoft Fabric, including OneLake, Lakehouse, Warehouse, Data Pipelines/Data Factory, notebooks and Dataflows Gen2.
- Strong SQL Server and T‑SQL capability, including complex query development, schema design, indexing and performance optimisation.
- Practical experience developing, maintaining and supporting production ETL/ELT pipelines.
- Experience integrating and extracting data from REST/SOAP APIs, databases, flat files and other structured or unstructured data sources.
- Proficiency in data transformation using SQL and Python/PySpark, with an understanding of scalable data processing practices.
- Practical experience in data warehousing, dimensional modelling, incremental loading, orchestration and schema evolution.
- Experience with troubleshooting pipeline failures, data quality issues and performance bottlenecks, including the ability to restore service efficiently.
- Experience with source control, CI/CD and deployment practices using Git, Azure DevOps or equivalent tools.
- Experience supporting Power BI and other downstream analytical or reporting requirements.
- Demonstrated ability to take ownership of an existing technical environment with limited hand‑holding.
- Strong documentation, communication and stakeholder engagement skills.
Skills and Knowledge (essential)
- Microsoft Fabric: OneLake, Lakehouse, Warehouse, Data Pipelines/Data Factory, notebooks and Dataflows Gen2.
- SQL Server / T‑SQL
- Python / PySpark
- REST/SOAP APIs and structured/unstructured data ingestion.
- ETL/ELT, incremental loading, orchestration and scheduling.
- Dimensional modelling, medallion architecture, schema evolution and data warehousing.
- Power BI integration and understanding of downstream analytical requirements.
- Git / Azure DevOps, CI/CD and environment deployment practices.
- Monitoring, data quality, performance optimisation, security and operational support.
Other
- Proficient in Afrikaans and English.
- Own transport and valid drivers license.
Key Responsibilities
- Take ownership of the existing Microsoft Fabric data engineering environment and become productive quickly following handover.
- Design, develop, maintain and orchestrate reliable batch and near‑real‑time data ingestion pipelines using appropriate Microsoft Fabric capabilities.
- Extract and ingest data from structured and unstructured sources, including REST APIs, SOAP APIs, databases and flat files.
- Develop robust data transformation logic using SQL, Python/PySpark, Fabric notebooks and Dataflows Gen2, as appropriate.
- Implement incremental loading, retry mechanisms, logging, monitoring and alerting to support data integrity and pipeline reliability.
- Troubleshoot and resolve pipeline failures and data processing issues efficiently.
- Optimise data pipelines and processing workloads for performance, scalability and cost‑effectiveness.
Benefits
- Competitive salary and comprehensive benefits package.
- Opportunities for professional development and training in Randburg.
- A hybrid work arrangement offering flexibility.
- Work on challenging projects that enable cutting‑edge AI advancements.
- Join a collaborative and innovative engineering team.
We are an equal opportunity employer: qualified applicants are considered for and treated during employment without regard to race, color, creed, religion, sex, national origin, citizenship status, disability status, protected veteran status, age, marital status, sexual orientation, gender identity, genetic information, or any other characteristic protected by law.