Senior Data Engineer
Summary
Designs and builds scalable data pipelines and reusable components using AWS, Spark, and Python to enable data democratization in a Data Mesh architecture.
Responsibilities
- Participate in understanding business needs to ensure value delivery
- Design architecture and evaluate available technologies to solve problems
- Implement data pipelines for data democratization within a Data Mesh architecture
- Develop and deliver reusable components such as connectors, APIs, SDKs, and CLIs
- Test and automate processes wherever possible
- Validate and adapt components based on user feedback
- Collaborate on product quality and technological innovation
Requirements
- Large-scale data processing knowledge (Apache Spark, Hadoop, Hive, etc.)
- Experience with AWS Data & Analytics services (Glue, EMR, Athena, Quicksight)
- Proficiency in Python
- Mastery of AWS Cloud services (S3, Lambda, SQS, DynamoDB, API Gateway)
- Practical experience with relational (SQL) and NoSQL databases
- Proficiency in DevOps practices and Infrastructure as Code (Terraform, CodePipeline, etc.)
Preferred Qualifications
- Previous experience with Data Mesh initiatives
- Experience with Generative AI or automation tools applied to data pipelines
- Knowledge of Observability tools (Cloudwatch, Prometheus, Grafana, etc.)
- Experience developing solutions for financial institutions
About the Company
Zup Innovation uses technology to transform lives, empower talent, and position underprivileged regions as high-quality technology hubs.