Data Engineer (Databricks Specialist)
Summary
Build and optimize scalable data pipelines on Databricks to ingest cybersecurity logs and network data, curating high-quality datasets for analytics using PySpark, Spark SQL, and Delta Lake.
Build scalable data pipelines and ensure curated, high-quality cyber data is available for analytics.
Key Responsibilities
- Build ingestion pipelines (logs, network data, security tools)
- Implement Delta Lake (Bronze-Silver-Gold layers)
- Optimise performance and cost (partitioning, caching)
- Ensure data quality, lineage, and reliability
Required Skills
- Strong PySpark / Spark SQL
- Experience with streaming (Kafka / Event Hub)