Senior/Principal Data Engineer - Financial Services/Banking

We are looking for an experienced Senior / Principal Data Engineer to design, build, and optimize scalable data platforms and infrastructure supporting advanced analytics, AI, and Large Language Model (LLM) applications.

The ideal candidate combines strong data engineering expertise with hands-on experience in cloud infrastructure, AI/LLM data pipelines, and modern security and performance technologies, including Cloudflare. This role will play a key part in developing reliable, secure, and high-performance data systems that support business intelligence, machine learning, and AI-driven products.

Key Responsibilities

1. Data Architecture & Engineering

• Design, develop, and maintain scalable data pipelines, ETL/ELT workflows, and data processing systems.
• Build and optimize data architectures, including data lakes, data warehouses, and lakehouse solutions.
• Ensure data quality, consistency, reliability, governance, and availability across multiple systems.
• Optimize data processing performance, storage efficiency, and infrastructure costs.

2. AI & LLM Integration

• Develop data pipelines to support LLM applications, including data ingestion, preprocessing, transformation, and indexing.
• Build and maintain data infrastructure for Retrieval-Augmented Generation (RAG), vector databases, embeddings, and semantic search.
• Prepare and optimize structured and unstructured data for AI model training, fine-tuning, and inference.
• Collaborate with AI/ML engineers and product teams to operationalize LLM-powered applications.
• Implement data evaluation, monitoring, and quality controls to improve the reliability of AI systems.
• Design and maintain secure, scalable, and highly available cloud-based data infrastructure.
• Integrate Cloudflare services into application and data platform architectures where appropriate.
• Leverage relevant Cloudflare capabilities, such as CDN, WAF, API Shield, Zero Trust, and Workers, to improve security, performance, and API connectivity.
• Implement monitoring, access controls, network security, and performance optimization across data services.
• Collaborate with platform and DevOps teams to establish infrastructure automation and deployment best practices.

4. Technical Leadership & Collaboration

• Lead technical design discussions and establish data engineering standards and best practices.
• Mentor engineers, review code, and guide the implementation of complex data solutions.
• Work closely with software engineering, AI/ML, product, security, and business stakeholders.
• Drive technical decision-making, architecture reviews, and continuous platform improvements.

Requirements

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent practical experience.
• Minimum 5–8 years of experience in data engineering, software engineering, or data platform development, with demonstrated ownership of complex technical projects.
• Strong proficiency in Python and SQL, with experience in data processing frameworks such as Apache Spark, Kafka, or equivalent technologies.
• Experience designing and operating modern data platforms, including data warehouses, data lakes, or lakehouse architectures.
• Hands-on experience integrating LLMs into production systems, particularly data preparation, embeddings, RAG pipelines, vector search, or AI data infrastructure.
• Practical experience with cloud platforms such as AWS, Google Cloud Platform, or Microsoft Azure.
• Experience with Cloudflare products or technologies, particularly API security, WAF, Workers, CDN, or Zero Trust.
• Strong understanding of distributed systems, APIs, data security, observability, and production reliability.
• Experience with CI/CD, Infrastructure as Code, containerization, and automated testing is an advantage.
• Strong problem-solving, system design, communication, and stakeholder management skills.

Additional Qualifications for Principal-Level Candidates

• Proven track record of defining data platform architecture and making high-impact technical decisions.
• Experience leading cross-functional engineering initiatives and mentoring senior engineers.
• Ability to translate business and AI product requirements into scalable technical solutions.
• Experience establishing engineering standards, reliability metrics, and long-term technology roadmaps.

Preferred Skills

• LLM application architecture and RAG implementation.
• Vector databases such as Pinecone, Weaviate, Milvus, or pgvector.
• Orchestration tools such as Airflow, Dagster, or equivalent.
• Cloudflare Workers, API Shield, WAF, and Zero Trust.
• Kubernetes, Docker, Terraform, and cloud-native infrastructure.
• Data governance, privacy, lineage, and observability.