Lead/Senior Data Engineer

Our client is seeking an experienced, hands-on Senior Data Engineer to help build and scale the enterprise data foundation supporting Analytics, AI, and self-service decision-making across the company.

This role requires more than just moving data between systems. The successful candidate will act as a data product architect, transforming complex, messy operational data into trusted, reusable, business-ready data products. These products will power enterprise analytics, Tableau reporting, Databricks Genie / Genie Code, machine learning, and emerging GenAI use cases across Manufacturing, Supply Chain, Sales, Finance, and HR.

You will play a pivotal role in shifting the organization away from one-off, siloed pipelines toward a highly governed, centralized single-source-of-truth platform built entirely on Databricks and AWS.

Key Responsibilities:

1. Data Engineering & Lakehouse Development

• Design, develop, test, and maintain scalable ETL/ELT pipelines using Databricks, Spark, SQL, Python, and AWS.
• Build and maintain lakehouse solutions leveraging Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
• Integrate structured and semi-structured data from enterprise applications, manufacturing systems, databases, APIs, and cloud operational sources.
• Optimize Databricks workloads, SQL queries, Delta tables, compute utilization, and end-to-end pipeline performance.
• Implement orchestration, monitoring, error handling, data reconciliation, and automated recovery for production data pipelines.
• Design scalable enterprise data models (dimensional modeling, star schemas, fact/dimension structures) supporting cross-functional analytics and standardized KPIs.
• Transform raw operational data into curated, business-ready data products that can be reused across dashboards, analytics, AI, and ML applications.
• Collaborate closely with business and analytics teams to translate complex business definitions into technically robust data models.

3. Analytics AI & Self-Service Enablement

• Build AI-ready and Genie-ready data products that allow business users to securely analyze enterprise data using Databricks Genie and Genie Code.
• Structure data, metadata, relationships, business definitions, and semantic/contextual layers to maximize the accuracy of conversational AI analytics.
• Enable engineers, analysts, and business users to perform governed self-service analysis without creating uncontrolled copies of data.
• Partner with data science teams to prepare feature-ready datasets for predictive analytics, machine learning, and MLOps workloads.

4. Data Governance, Quality & DataOps

• Implement data governance, cataloging, lineage, metadata management, and role-based access controls using Databricks Unity Catalog.
• Build automated data-quality checks for completeness, accuracy, consistency, freshness, and cross-system reconciliation.
• Apply Git-based development, code reviews, testing, version control, and automated CI/CD processes for Databricks assets.
• Focus heavily on developing reusable engineering frameworks and templates rather than building hard-coded, one-off pipelines.