Senior Data Engineer

At cloudandthings.io, we are an engineering-led consultancy focused on building modern Data and AI Platforms for enterprise clients.

We don't just design architectures; we build real systems that power analytics, decision-making, and AI at scale. We operate with high standards, strong ownership, and a bias for execution.

We are looking for people who think from first principles, challenge conventional thinking, and take deep ownership of what they deliver - engineers who don't simply identify problems, but solve them and see them through to a running system in production.

About cloudandthings.io

At cloudandthings.io, we are an engineering-led consultancy focused on building modern Data and AI Platforms for enterprise clients.

We don't just design architectures; we build real systems that power analytics, decision-making, and AI at scale. We operate with high standards, strong ownership, and a bias for execution.

We are looking for people who think from first principles, challenge conventional thinking, and take deep ownership of what they deliver - engineers who don't simply identify problems, but solve them and see them through to a running system in production.

The Role

As a Senior Data Engineer, you will lead the design, build, and operation of modern, cloud-native data platforms across AWS and Azure, using Databricks, Microsoft Fabric, and cloud-native services. You will own delivery across the full data lifecycle — ingestion, transformation, modelling, and serving — enabling real-time analytics, reporting, and AI use cases.

This is a hands‑on senior role. You will still be in the code every day, but you will also be the person clients and engineers look to for technical direction: shaping the architecture, making the trade‑off calls, setting the standards, and raising the level of everyone around you.

Much of our work centres on lakehouse platforms, so proven depth in Databricks and/or Microsoft Fabric is essential.

Key Responsibilities

The list below is broad. We expect production‑grade experience across a substantial portion of it, and the judgement to know which tools a given problem actually calls for.

Technical Leadership and Delivery Ownership

• Act as technical lead on client engagements: own the solution design end to end, from discovery and architecture through to a supported production platform.
• Translate ambiguous business requirements into pragmatic designs, and communicate them credibly to engineers and senior client stakeholders.
• Make and justify architectural trade‑offs.
• Set and enforce engineering standards: code review, testing, branching and release strategy, definition of done.
• Mentor and grow junior and mid-level engineers through pairing, review and feedback; contribute to hiring and technical assessment.
• Contribute to internal capability: reusable accelerators, reference architectures, and solution‑shaping input.

Software Engineering Foundations

• Strong grounding in software engineering fundamentals (data structures, algorithms, design patterns), and the ability to hold others to that standard.
• Advanced proficiency in Python and SQL (additional languages advantageous).
• Deep experience with Git and CI/CD — able to design a branching and deployment strategy for a team, not just work within one.
• Hands‑on Infrastructure as Code with Terraform or Bicep, including the Databricks provider and Fabric deployment automation; able to own a platform's IaC estate.
• Promoting data assets as code: Databricks Asset Bundles and Repos, Fabric Git integration and deployment pipelines.
• Comfortable in Linux environments.

Ingestion and Streaming

• Design scalable ingestion across hybrid and cloud environments, and the standards other engineers will follow.
• Streaming: AWS Kinesis / MSK (Kafka), Azure Event Hubs, Databricks Structured Streaming, Fabric Eventstream and Real‑Time Intelligence (Eventhouse / KQL).
• Incremental and batch: Auto Loader, Lakeflow Connect, Fabric Dataflows Gen2 and Copy Job, AWS DataSync / DMS, Azure Data Factory / Synapse / Fabric Data Pipelines.
• Integration via APIs, JDBC/ODBC and CDC, including Fabric Mirroring (Azure SQL, Cosmos DB, Snowflake, Unity Catalog) and Delta Sharing.

Storage and Lakehouse

• Design and manage data lakes on Amazon S3 and Azure Data Lake Storage Gen2, and architect lakehouse platforms using:
• Databricks — Delta Lake, Unity Catalog (catalogs, schemas, external locations, storage credentials, volumes), managed vs external tables, streaming tables and materialised views, Lakehouse Federation, Delta Sharing.
• Microsoft Fabric — OneLake and shortcuts, Lakehouse, Warehouse, Fabric SQL Database, workspace and domain design, capacity (F SKU) sizing and CU management.
• Table optimisation: partitioning, liquid clustering, Z‑ordering, OPTIMIZE / VACUUM, V‑Order and statistics, small‑file management, time travel and retention.
• Open formats and standards: Delta Lake, Parquet, Avro, JSON, Apache Iceberg / Delta UniForm.
• Relational (Postgres, SQL Server, Aurora), NoSQL (DynamoDB, Cosmos DB) and caching (Redis).

Processing and Transformation

• Build scalable ETL/ELT pipelines using:
• Databricks — PySpark and Spark SQL, Delta Live Tables / Lakeflow Declarative Pipelines, Workflows, serverless and job compute, Photon, cluster policies, dbt.
• Microsoft Fabric — Spark notebooks and job definitions, environments and library management, Warehouse T‑SQL and stored procedures, Dataflows Gen2.
• AWS / Azure native — Glue, Lambda, EMR, Synapse Spark.
• Own the data model: medallion architecture (Bronze/Silver/Gold), dimensional models (star schemas, SCD handling), and platform‑wide modelling standards.
• Build reusable, testable pipelines: modular notebooks and Python packages, parameterisation, unit and integration testing, idempotent and restartable loads.
• Lead Spark performance and cost tuning: skew and shuffle management, caching, right‑sizing compute, autoscaling — and diagnose the problems others cannot.

Analytics and AI Enablement

• Design platforms that support BI, advanced analytics, machine learning and AI use cases:
• Databricks — SQL warehouses, AI/BI dashboards and Genie, MLflow, Unity Catalog model registry, Model Serving, Mosaic AI (Vector Search, agent and RAG patterns), feature engineering.
• Microsoft Fabric — semantic models and Direct Lake, Power BI report and dataset design, DAX, Fabric Data Science, Data Agents and Copilot.
• AWS — Redshift, Athena, SageMaker and Bedrock where relevant.
• Partner with data scientists and analysts on feature engineering and real‑time decisioning systems.
• Establish data quality, observability and lineage frameworks: DLT expectations, Unity Catalog lineage and system tables, Fabric monitoring hub, freshness and volume alerting.

Security, Governance and Compliance

• Design secure, enterprise‑grade platforms: AWS IAM, Azure Entra ID, RBAC, managed identities and service principals.
• Governance: Unity Catalog (namespace design, grants and inheritance, row filters and column masks, attribute‑based access, audit logs); Fabric (workspace roles and item permissions, OneLake security, sensitivity labels, domains, Purview); AWS Lake Formation.
• Networking: VPC / VNets, private endpoints, Direct Connect / ExpressRoute, Databricks secure cluster connectivity and Private Link, Fabric private links and managed VNets.
• Encryption: KMS / Key Vault / TLS, customer‑managed keys, secret scopes.
• Engage with client security and architecture forums to get designs approved.

Orchestration and Operations

• Orchestrate with Databricks Workflows, AWS Step Functions / MWAA (Airflow), ADF / Synapse and Fabric Data Pipelines.
• Monitoring and observability: CloudWatch, Azure Monitor, Log Analytics, Databricks system tables and query history, Fabric Capacity Metrics and monitoring hub.
• Own the operational posture of what you build: SLAs, alerting, incident response, root cause analysis, and runbooks.
• Drive reliability, performance and cost optimisation (FinOps): compute policies, serverless vs classic, tagging and chargeback, Fabric capacity smoothing and pause/resume.

Requirements

• Bachelor's degree in Engineering, Computer Science, or a related field.
• Typically 7+ years in data or software engineering, with at least 3 years building data platforms on AWS and/or Azure.
• A track record of delivering production data solutions end‑to‑end — including at least one platform you can talk through in depth: the constraints, the decisions, and what you'd do differently.
• Substantial hands‑on production experience with at least one lakehouse platform (Databricks or Fabric), working knowledge of the other, and the appetite to build depth.
• Experience leading a workstream or small team: setting technical direction, reviewing work, and mentoring.
• Confident client‑facing communication. Able to run a design workshop, defend an architecture, and explain a trade‑off to a non‑technical audience.
• Certifications advantageous, professional or expert level especially: Databricks (Data Engineer Professional / Associate, Machine Learning Associate); Microsoft (DP‑700, DP‑600, DP‑203, AZ‑305); AWS (Data Engineer Associate, Solutions Architect Associate / Professional).
• Experience with Docker, Kubernetes and CI/CD for data workloads.
• Strong working knowledge of data security and compliance standards.
• Strong problem‑solving skills, with the judgement to know when a simple solution beats a sophisticated one.
• Self‑organising, able to prioritise across multiple tasks and competing stakeholders.
• Excellent written and verbal communication, and a collaborative approach with clients and colleagues.
• Willingness to travel to clients as and when required.

What We Offer

• A culture of engineering and an environment where ideas are heard, and builders can build.
• Competitive compensation and bonus structure.
• A flexible, supportive environment that values diversity, work‑life balance and personal growth.
• Career advancement, mentorship and ongoing development.
• Cutting‑edge technology, engaging client projects, and a talented team around you.