Lead Data Engineer

Lead Data Engineer

Role Purpose

Exp : 8+yrs to 15 yrs

Location : Chennai

Np : Immediate to 30days

Required Skills

Databricks (Highest Priority)

· Databricks

• PySpark
• Spark SQL
• Delta Lake
• Unity Catalog
• Databricks Workflows
• Databricks Asset Bundles
• Lakehouse Architecture

Azure Platform

· Azure Data Factory (ADF)

• ADLS Gen2
• Azure SQL
• Azure Security Patterns

Migration Experience

· Synapse Pipelines

• Synapse Spark
• Synapse SQL Pools
• Synapse Notebooks
• Synapse Data Flows
• Modernization of legacy ETL workloads

Engineering

· Advanced SQL

• Data Modeling
• GitLab CI/CD
• Performance Tuning
• Code Reviews

Preferred

· Apache Airflow

• Snowflake
• Data Governance

Enterprise-scale Azure platform implementation

͏

Areas of responsibility

͏

Gathering of requirements-Assist in conducting workshops and discussions with client stakeholders to gather business process requirements.

Coding and Configuration-"Develop and deliver code for assigned modules, in alignment with client requirements ensuring proper integration with system components.

͏

Testing and Trial Runs-"Execute testing for assigned modules, validate functionality across required interactions, and participate in User Acceptance Testing (UAT) sessions.

Implementation-Participate in implementation of software application and its integration with other systems to ensure stability.

Maintain Documentation-Prepare detailed reports and documentation on project specifications, activities, and status, while presenting information using flowcharts, layouts, diagrams, code comments, and clear, well-structured code.

͏

Stakeholder Collaboration-Collaborate with cross functional teams to understand requirements, support development activities, check system compatibility, and ensure solutions meet expected standards.