Lead Data Platform Engineer (Contract)

We are currently recruiting for a Lead Data Platform Engineer to join a programme focused on data platforms, analytics engineering, and modern Lakehouse solutions. The client is looking for a hands‑on engineering professional with strong Microsoft Fabric, Spark, PySpark and SparkSQL expertise, capable of designing, building and supporting enterprise‑scale data platforms.

Start Date: ASAP
Duration: 3 Months Initially
Pay Rate: Competitive
IR35 Status: Outside IR35
Location: London (1 day per week onsite, remainder remote)

The Role:

The core requirement is strong Lakehouse expertise, particularly Spark, PySpark, and SparkSQL. Hands‑on experience in low‑code is essential.

Data & Analytics Expertise:

• Data architecture (Lakehouse, Medallion, Kimball, 3NF).
• Data ingestion and processing.
• Data quality frameworks (Great Expectations, Soda).
• Data, BI and analytics platforms:
  • Microsoft Fabric
  • Purview
  • APIs
  • Streams
• Data governance and data literacy.
• KPI and semantic model design.
• AI and innovation in analytics.

Software Engineering:

Strong knowledge of:

• Design, Build, Test, Release, Deploy using Azure Fabric, data engineering tools.
• Secure coding practices.
• API design and integration patterns.
• Integrations using APIs, ADF for managing wide variety of data from structured, unstructured to semi.
• CI/CD pipelines and automated deployments.
• Infrastructure as Code (IaC):
  • Terraform
  • Git actions
• Containerisation:
  • Docker
• Security practices:
  • Identity and access management
  • Secrets management

Agile Delivery:

Strong experience in:

• Agile, Scrum, and Lean methodologies.
• Backlog management and roadmap planning.
• Technical cross team dependency management.
• Continuous improvement of delivery performance.

Development Tools:

Hands‑on capability with:

• Pyspark, pytest, DAX, python, Azure functions, Lamda, and SQL.
• Build and dependency management tools.
• CI platforms.
• Terraform/Ansible (or similar).

Platform‑Specific Expertise (Nice to have):

The role expects deep expertise in at least one of:

• APIs & Integrations
• QA Automation (Playwright, Selenium)
• Reporting & Analytics

Technical Leadership & Architecture (Nice to have):

• Lead technical direction for products, platforms, and services.
• Define and promote software development standards, patterns, and governance.
• Drive architectural consistency and manage technical debt.
• Review, establish and improve product software development lifecycles.