Data Engineer

Key Responsibilities

• Design and implement ETL/ELT pipelines using Spark SQL and Python within Databricks Medallion architecture

• Develop dimensional data models following star schema methodology with proper fact and dimension table design, SCD implementation, and optimization for analytical workloads

• Optimize Spark SQL and DataFrame operations through appropriate partitioning strategies, clustering and join optimizations to maximize performance and minimize costs

• Build comprehensive data quality frameworks with automated validation checks, statistical profiling, exception handling, and data reconciliation processes

• Establish CI/CD pipelines incorporating version control, automated testing including but not limited to unit test, integration test, smoke test, etc.

• Implement data governance standards including row-level and column-level security policies for access controls and compliance requirements

• Create and maintain technical documentation including ERDs, schema specifications, data lineage diagrams, and metadata repositories

  
  

Key Requirements

• Min. 4 years’ fulltime experience with SparkSQL, Python and PySpark for data engineering workflow

• Strong proficiency in dimensional modeling and star schema design for analytical workloads

• Experience implementing automated testing and CI/CD pipelines for data workflows

• Familiarity with GitHub operations and collaborative development practices

• Demonstrated ability to optimize engineering workflow jobs for performance and cost efficiency

• Experience with cloud data services and infrastructure (AWS, Azure, or GCP)

• Proficiency with IDE tools such as Visual Studio Code for efficient development

• Experience with Databricks platform will be a plus