Senior Data Engineer

Required Skills & Technical Expertise

• SQL High: Strong expertise in writing, debugging, and optimizing SQL queries.
• Python / PySpark – High: Hands-on experience in data processing, transformation, and pipeline development.
• Spark on YARN – High: Strong understanding of Spark job execution, monitoring, and troubleshooting in YARN environments.
• Hive / Data Modelling – High: Experience with Hive tables, data structures, and data warehouse modelling.
• Scheduling & Orchestration – Medium: Familiarity with Piper or similar scheduling and orchestration tools.
• Data Governance & Security – Medium: Understanding of PII handling, data privacy, and deletion workflows.
• Data Quality & Validation – Medium: Experience implementing validation checks and resolving data discrepancies.
• File-Based Ingestion – Medium: Experience with SFTP, batch CSV ingestion, and file-processing workflows.
• Data Infrastructure & Architecture – Medium: Understanding of data pipeline architecture and infrastructure components.