Data Engineer - AWS Pipelines
Description
About the Role :
As a Data Engineer, you will own the design and delivery of robust AWS data pipelines with deep LLM integration.
What You'll Do :
- Design, build, and maintain scalable AWS data pipelines using S3, Glue, Redshift, Lambda, and related services.
- Integrate LLM capabilities into data workflows for enrichment, classification, and summarization.
- Collaborate with AI/ML teams to ensure data readiness for model training and inference.
- Build monitoring, quality checks, and alerting for production data pipelines.
- Optimize query performance and data storage for cost efficiency.
- Work with cross-functional stakeholders to understand data requirements and deliver solutions.
Must Have :
- AWS Data stack - S3, Glue, Redshift, EMR, Lambda, Step Functions.
- Strong SQL and Python/PySpark for data transformation.
- Experience with data pipeline orchestration (Airflow, Step Functions, or similar).
- LLM integration experience - using LLM APIs within data processing workflows.
Good to Have :
- AWS Data certifications.
- Streaming data experience (Kinesis, Kafka).
- Exposure to MLOps and ML data pipelines.