Data Engineer / DevOps Engineer
Posted Updated
Summary
Builds and maintains enterprise-scale data pipelines on Databricks using Medallion architecture, Delta Lake, and Spark/PySpark, including real-time ingestion via Structured Streaming. Also tunes clusters and costs, enforces governance through Unity Catalog, and monitors pipeline health with alerts and dashboards.
Experience Requirements
- Total Data Engineering: 5+ years
- Databricks Specific: 1+ years of hands-on experience in production environments
- Enterprise Scale: Experience building enterprise-level scalable pipelines with Spark or Hadoop-based systems
Required Certifications
- Mandatory: Databricks Certified Data Engineer Professional.
- Highly Preferred: Apache Spark Developer Associate
- Cloud (Optional): Professional-level certification in AWS.
Key Responsibilities
- Design & Build: Develop robust Medallion Architecture (Bronze/Silver/Gold) pipelines using Lakeflow and Delta Lake
- Performance Tuning: Proactively monitor and tune clusters, jobs, and serverless compute to reduce latency and infrastructure costs
- Governance Integration: Ensure every data asset is correctly tagged, secured through ACL, and discoverable via Unity Catalog
- Code Review: Enforce software engineering best practices (unit testing, modular code, and documentation) within the data team
- Streaming & Ingestion: Implement real-time or near-real-time data ingestion using Structured Streaming and Lake flow Connect
- Observability: Monitor pipeline health and costs using system tables; set up alerts and dashboards for proactive issue detection.
Preferred Candidate Background
- Distributed Systems: Proven track record with Open-Source Apache Spark, Flink, or Hadoop.
- Python-Centric: Advanced Python skills are a prerequisite; the project requires building complex logic that exceeds standard SQL capabilities.
- Engineering Mindset: Experience in refactoring legacy Big Data workloads into clean, testable, and optimized PySpark applications.