Senior Cloud Data Engineer
Posted Updated
Description
Position Description:
.
Job Title: Cloud Data Engineer
Position: Senior Software Engineer
Experience: 6-10 Years
Category: Software Development/ Engineering
Shift: Hybrid - (1PM to 10 PM Shift)
Main location: Hyderabad
Position ID: J0826-0434
Employment Type: Full Time
Education Qualification: Bachelor's degree in Computer Science or related field or higher with minimum 6 years of relevant experience.
Position Description:We are looking for an experienced Senior Cloud Data Engineer with strong expertise in AWS cloud services, Python/PySpark, SQL, Databricks, and data engineering. The ideal candidate will be responsible for designing, developing, and maintaining scalable cloud-native data solutions, secure APIs, and efficient ETL/ELT pipelines. The role requires hands-on experience in cloud architecture, API development and integration, Infrastructure as Code (IaC), and DevOps practices to deliver secure, reliable, and high-performance data platforms.
Job Title: Cloud Data Engineer
Position: Senior Software Engineer
Experience: 6-10 Years
Category: Software Development/ Engineering
Shift: Hybrid - (1PM to 10 PM Shift)
Main location: Hyderabad
Position ID: J0826-0434
Employment Type: Full Time
Education Qualification: Bachelor's degree in Computer Science or related field or higher with minimum 6 years of relevant experience.
Position Description:We are looking for an experienced Senior Cloud Data Engineer with strong expertise in AWS cloud services, Python/PySpark, SQL, Databricks, and data engineering. The ideal candidate will be responsible for designing, developing, and maintaining scalable cloud-native data solutions, secure APIs, and efficient ETL/ELT pipelines. The role requires hands-on experience in cloud architecture, API development and integration, Infrastructure as Code (IaC), and DevOps practices to deliver secure, reliable, and high-performance data platforms.
Your future duties and responsibilities:
- Design, develop, and maintain scalable data engineering solutions using AWS cloud services, Databricks, Python, PySpark, and SQL.
- Build, optimize, and manage robust ETL/ELT pipelines for data ingestion, transformation, processing, and integration.
- Develop and integrate secure, scalable REST APIs using AWS API Gateway, AWS Lambda, and other cloud-native services.
- Implement API security mechanisms, including OAuth 2.0, JWT, IAM policies, and appropriate access controls.
- Design and manage cloud infrastructure using AWS services such as S3, ECS/EKS, CloudFront, ALB/NLB, KMS, and IAM.
- Develop and maintain Infrastructure as Code using Terraform to automate infrastructure provisioning and configuration.
- Design cloud-native applications and data solutions, following software architecture principles, design patterns, scalability, and high availability best practices.
- Optimize data processing workloads, SQL queries, and PySpark jobs to improve performance, scalability, and cost efficiency.
- Implement secure data storage, encryption, access management, and other cloud security best practices.
- Collaborate with architects, developers, DevOps teams, and business stakeholders to translate requirements into technical solutions.
- Establish and maintain CI/CD pipelines for automated build, testing, deployment, and release management.
- Troubleshoot production issues, perform root cause analysis, and implement preventive measures to improve system reliability.
- Develop and maintain technical documentation, API specifications, architecture diagrams, and deployment procedures.
Required qualifications to be successful in this role:
Must-Have Skills
- AWS Cloud Services: Hands-on experience with AWS Lambda, API Gateway, S3, ECS/EKS, CloudFront, ALB/NLB, IAM, and KMS.
- Python / PySpark: Strong programming skills in Python and experience developing scalable data processing solutions using PySpark.
- SQL: Strong SQL skills, including complex queries, joins, aggregations, and query optimization.
- Databricks: Experience developing and optimizing data processing workloads and ETL/ELT pipelines using Databricks.
- Data Engineering: Strong understanding of data ingestion, transformation, data integration, and pipeline development.
- API Development Integration: Experience designing, developing, consuming, and integrating RESTful APIs.
- API Security: Hands-on knowledge of OAuth 2.0, JWT, authentication, authorization, and secure API implementation.
- Infrastructure as Code: Practical experience with Terraform for infrastructure provisioning and management.
- Cloud-Native Development: Understanding of cloud-native application design, scalability, resilience, and distributed systems.
- Software Architecture Design: Knowledge of application design principles, design patterns, modular architecture, and performance optimization.
Good-to-Have Skills
- Experience with OpenAPI / Swagger for API documentation, specification, and contract validation.
- Hands-on experience with CI/CD tools such as Jenkins, GitHub Actions, and GitLab CI.
- Knowledge of DevOps practices, infrastructure automation, and configuration management.
- Experience with Docker, containerization, Kubernetes orchestration, and deployment automation.
- Experience working in Agile/Scrum environments and collaborating with cross-functional engineering teams.
Skills:
- AWS Lambda
- AWS S3
- Cloud Native Development
- Databricks
- English
- Infrastructure as a Code
- SQL
- AWS Cloudformation
- Data Engineering
- Data Engineering
- Python
Skills
- Agile
- API
- Api Documentation
- API Gateway
- Api Security
- Authentication
- Automation
- AWS
- CI/CD
- Cloud
- Cloud Native
- Cloud Security
- CloudFormation
- CloudFront
- Containerization
- Data Engineering
- Data Ingestion
- Databricks
- Design Patterns
- DevOps
- Distributed Systems
- Docker
- ECS
- EKS
- ELT
- ETL
- GitHub
- GitHub Actions
- GitLab
- IAM
- Infrastructure as Code
- Jenkins
- JWT
- Kubernetes
- Lambda
- OAuth
- OpenAPI
- PySpark
- Python
- REST
- Root Cause Analysis
- S3
- Scrum
- SQL
- Swagger
- Terraform