Data Support Engineer
A Data Application Support role specializing in Databricks focuses on maintaining, troubleshooting, and optimizing Spark-based data pipelines and Lakehouse architectures. Key responsibilities include resolving L2/L3 production incidents, performance tuning SQL/Python (PySpark) jobs, managing Delta Lake assets, and collaborating with data engineers to ensure data reliability.
Key Responsibilities
- Incident & Problem Management: Provide L2/L3 support for data applications, resolving production issues and troubleshooting Databricks jobs, notebooks, and workflows.
- Performance Tuning: Optimize Spark applications, SQL queries, and Delta Lake tables to improve efficiency and reduce costs.
- Pipeline Maintenance: Monitor and troubleshoot ETL/ELT pipelines in Databricks (including Data Factory/Delta Live Tables), ensuring data quality and lineage (Unity Catalog).
- Collaboration: Act as a liaison between users, data engineering teams, and platform engineering, providing technical expertise and contributing to documentation.
- Automation: Create tools to automate routine support tasks and enhance support team productivity. Databricks +5
Required Qualifications & Skills
- Technical Expertise: Strong hands-on experience with Apache Spark, Python/PySpark, and SQL.
- Databricks Ecosystem: Proficiency with Databricks Unified Data Analytics Platform, Delta Lake, and ideally Azure Databricks.
- Cloud Data Storage: Experience with Azure Data Lake Storage (ADLS Gen2) or similar data lake technologies.
- Version Control & CI/CD: Experience with Git/Azure DevOps for code management.
- Problem-Solving: Strong analytical skills, with the ability to diagnose complex data processing bottlenecks.
"Nice-to-Have" Skills
- Data orchestration tools (e.g., Apache Airflow, Azure Data Factory).
- Data governance tools (e.g., Unity Catalog, Collibra).
- Streaming data knowledge (e.g., Spark Structured Streaming).