Data Engineer

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and platform solutions across AWS cloud and hybrid/on-premises environments.

• Build and optimize ETL/ELT workflows using Python, PySpark, SQL, and cloud-native technologies.

• Develop and support data lakes, lakehouse architectures, metadata management, search platforms, and AI-enabled services.

• Implement and maintain CI/CD pipelines, automated testing frameworks, Infrastructure as Code (IaC), and deployment processes.

• Integrate and support AWS services and platform technologies, including OpenSearch, Amazon Bedrock, AgentCore, and Model Context Protocol (MCP).

• Monitor and optimize platform reliability, security, performance, observability, and cloud costs.

• Troubleshoot production issues, perform root cause analysis, and ensure platform stability and operational excellence.

• Collaborate with cross-functional teams to drive platform modernization, automation, and continuous improvement.

Mandatory Requirements

• 3–5+ years of hands-on experience developing and supporting production-grade data pipelines using SQL, Python, and cloud technologies.

• Practical experience with AWS cloud services and/or hybrid cloud environments.

• Strong Linux/Unix administration, shell scripting, and production support capabilities.

• Solid understanding of data engineering principles, distributed processing, and enterprise platform operations.

Technical Skills

• Data Engineering: ETL/ELT, data lakes, lakehouse architecture, distributed data processing, SQL, Python, PySpark, and Apache Spark.

• AWS & Cloud Platforms: AWS Glue, EMR, S3, Redshift, Lambda, and Amazon Bedrock.

• Data & Streaming Technologies: Apache Airflow, dbt, Kafka/MSK, Apache Iceberg, Hive, Trino, and Presto.

• Search & AI Platforms: OpenSearch, AgentCore, MCP, and AI service integration.

• DevOps & Platform Engineering: Git, CI/CD, DevOps practices, Infrastructure as Code (IaC), and automated testing.

• Monitoring & Observability: CloudWatch, Datadog, Grafana, and Prometheus.

Preferred Qualifications

• Experience building and supporting data lake and lakehouse platforms.

• Knowledge of streaming pipelines and event-driven architectures.

• Hands‑on experience with OpenSearch administration, performance optimization, and tuning.

• Familiarity with monitoring, logging, alerting, and observability tools.

• Understanding of data governance, security, access controls, and Personally Identifiable Information (PII) protection.

• Experience supporting enterprise platforms within banking, insurance, or financial services environments.