Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Astra-North Infoteck Inc.

NewBe an early applicant

Principal Databricks Data Engineer

Posted Updated

Summary

Principal-level data engineer (12–18 yrs experience) leading Cloudera-to-Databricks Lakehouse modernization for finance and risk data platforms — building PySpark/Spark SQL ELT pipelines, semantic/reporting layers, and governing data quality, lineage, and security on AWS or Azure, while mentoring senior engineers.

  • 12–18 years of overall Data Engineering experience
  • 8+ years of experience with enterprise Data Warehouse and Data Lake platforms
  • 5+ years of hands‑on experience with Databricks and Apache Spark at scale
  • Strong experience modernizing legacy Cloudera platforms including:
    • Modernize Cloudera platforms to Databricks Lakehouse architecture
    • Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to Cloud Object Storage and Delta Lake
    • Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines
    • Ensure data reconciliation, audit integrity, and consistency during migration
    • Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
    • Curated / Conformed Layer
    • Semantic / Consumption Layer
    • Modernize traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures
  • Strong experience with finance and risk data models including:
    • Financial Hierarchies
    • Liquidity Risk
  • Enable reporting capabilities including:
    • Build and manage semantic and consumption layers for BI, reporting, and analytics
    • Define business metrics, dimensions, hierarchies, and KPIs
  • Experience with:
    • Develop and optimize large‑scale data pipelines using:
    • Optimize workloads using:
    • Data Quality Frameworks
    • Establish data lineage and metadata management
    • Ensure data security, access control, and compliance standards
    • Experience with AWS or Azure cloud platforms
    • Experience with CI/CD pipelines using:
    • Familiarity with:
    • Experience with dbt is an advantage
  • Act as a technical authority and lead enterprise architecture decisions
  • Mentor senior engineers and establish engineering standards
  • Collaborate with finance, risk, analytics, and governance stakeholders
  • Translate complex data structures into business‑ready insights

Nice‑to‑Have Skills

  • Experience in BFSI, Capital Markets, or Regulatory Reporting
  • Oracle Financials
  • Experience supporting AI/ML workloads
  • Databricks and Cloud Certifications

Key Responsibilities

  • Lead Cloudera to Databricks transformation initiatives
  • Shape enterprise finance and risk data platforms
  • Support regulatory, management, and analytical reporting systems

Skills

What Principal Data Engineering jobs ask for — and how much of it you have →
Apply

See also

Data Engineering jobs by country — openings, pay and top skills →