DATA ENGINEER

We are seeking an On-site Senior Data Engineer who would be responsible for designing, building, and maintaining scalable, secure, and high-performance data infrastructure that powers analytics, AI/ML models, and enterprise applications. The role sits at the intersection of data engineering, applied machine learning support, and software systems, working closely with the Senior AI & Software Manager to translate product, AI, and business requirements into robust data pipelines and platforms.

This role is delivery-focused and impact-driven, with strong ownership of data reliability, performance, and governance across cloud and distributed environments.

• Design, develop, and maintain end-to-end ETL/ELT pipelines for structured and unstructured data using Python and SQL.
  

• Build scalable batch and near-real-time data workflows leveraging Apache Spark, Hadoop, Kafka, and Airflow.
  

• Implement data ingestion, transformation, validation, and enrichment pipelines across multiple data sources (APIs, files, databases, streaming systems).
  

• Ensure high data quality through automated checks, anomaly detection, and validation logic, including ML-assisted data quality monitoring.
  

Cloud Data Platforms & Warehousing

• Architect and manage cloud-based data solutions across AWS (S3, Glue, Redshift, EMR), GCP (BigQuery, Dataflow, Pub/Sub), and Azure (Data Factory).
  

• Design and optimize data warehouses and analytical data models to support BI tools, AI workflows, and operational analytics.
  

• Implement cost-efficient storage and compute strategies while maintaining performance and scalability.
  

AI & Machine Learning Enablement

• Work closely with the Senior AI & Software Manager to prepare, structure, and optimize datasets for machine learning and predictive analytics.
  

• Support ML pipelines by enabling feature engineering, training data generation, and inference-ready data flows.
  

• Collaborate on integrating ML outputs into production systems and dashboards.
  

• Ensure data pipelines align with AI model requirements for freshness, latency, and reliability.
  

Software & API Integration

• Develop and maintain data services and APIs using FastAPI, Django REST, or Flask to expose data to applications and AI systems.
  

• Collaborate with software engineers to integrate data pipelines into broader system architectures.
  

• Ensure data platforms align with software engineering best practices (modularity, versioning, CI/CD readiness).
  

Analytics, Reporting & Decision Support

• Enable downstream analytics and reporting through clean, well-modeled datasets.
  

• Support BI and visualization tools such as Power BI and Looker by delivering optimized datasets and semantic layers.
  

• Partner with stakeholders to translate analytical and operational needs into technical data requirements.
  

Governance, Security & Compliance

• Implement data governance standards, access controls, and compliance measures, particularly for sensitive or regulated datasets.
  

• Ensure data integrity, traceability, and auditability across pipelines and storage layers.
  

• Collaborate on defining data documentation, lineage, and metadata practices.
  

Collaboration & Leadership

• Act as a senior technical partner to the Senior AI & Software Manager, contributing to architectural decisions and system design discussions.
  

• Collaborate with data scientists, AI engineers, software developers, and non-technical stakeholders.
  

• Provide technical guidance and mentorship to junior data engineers or analysts when required.
  

• Participate in planning, estimation, and delivery of complex data-driven projects.
  

Requirements

• Strong proficiency in Python and SQL for data engineering and analytics.
  

• Hands-on experience with Apache Spark, Hadoop, Kafka, and Airflow.
  

• Solid understanding of ETL/ELT design patterns, data modeling, and warehousing.
  

• Experience with cloud data platforms (AWS, GCP, Azure).
  

• Familiarity with machine learning workflows, including data preparation and feature engineering.
  

• Experience building APIs and services using FastAPI, Django REST, or Flask.
  

• Working knowledge of Docker, Kubernetes, and Git.
  

• Experience supporting BI tools such as Power BI or Looker.
  

Professional Experience

• Proven experience delivering large-scale, production-grade data systems.
  

• Experience working on multi-stakeholder, high-impact projects, including government or enterprise environments.
  

• Demonstrated ability to reduce processing time, improve data quality, and scale data operations.
  

• Track record of translating business or AI requirements into reliable technical solutions.
  

Education & Background

• Degree in Engineering, Computer Science, Statistics, or a related technical field.
  

• Formal training or certification in Data Science, Big Data, or Machine Learning is a strong advantage.