Data Engineer (Spark)
Posted 3
views
Summary
Data engineer who designs, builds, and maintains scalable data pipelines, models, and integrations for a new CRM platform, enabling customer insights, matching, and reporting. Core stack: Apache Spark, AWS (S3, RDS/PostgreSQL, IAM, EMR), SQL, and ETL/ELT tooling with a focus on data quality and governance.
We are seeking a Data Engineer to support the implementation of a new CRM platform by designing, building, and maintaining scalable data pipelines and integrations. The role will play a key part in enabling customer insights, opportunity matching, reporting, analytics, and operational processes while ensuring data quality, governance, and scalability.
Key Responsibilities
- Analyse existing data sources and define data contracts with business consumers.
- Design, develop, and maintain data ingestion, transformation, and publishing pipelines.
- Build data models supporting customer profiles, opportunities, matching engines, tracking, and reporting.
- Implement data quality controls, reconciliation processes, and monitoring.
- Optimise ETL/ELT processes for performance, scalability, and cost efficiency.
- Work closely with Full Stack Developers to expose and consume data within the application.
- Document data lineage, models, business rules, dependencies, and recovery procedures.
- Ensure compliance with security, governance, and privacy standards.
Required Skills
- Strong experience with Apache Spark for large-scale data processing.
- Hands‑on experience with AWS, including S3, RDS/PostgreSQL, IAM, and EMR.
- Solid knowledge of SQL for transformation, optimisation, and analytics.
- Experience designing and implementing ETL/ELT pipelines.
- Strong understanding of data modelling, data quality, metadata management, and traceability.
- Experience with Git, CI/CD, automated testing, monitoring, and alerting.
- Knowledge of security principles including IAM, encryption, secrets management, and least‑privilege access.
- Experience working in Agile delivery environments.