Data Engineer
Summary
Data Engineer in Hyderabad who builds and maintains data pipelines using AWS Glue, Python, and PySpark, develops SQL-based transformations loading data into Snowflake, implements data quality checks, monitors production pipelines, and documents workflows. Terraform is used for infrastructure provisioning under supervision.
Key Responsibilities and Duties
- Formulates system scope through research and fact-finding to develop or modify moderately complex information systems.
- Evaluates and implements changes to existing system logic difficulties and documentation as necessary.
- Reviews flow charts, models, and procedures and feasibility studies to design possible system solutions.
- Reviews technical documentation to guide system users and to assist with the ongoing operation, maintenance, and development of the system.
- Collaborates with system analysts, engineers, and programmers to design systems and to determine project limitations and capabilities, performance requirements, systems performance standards, and interfaces.
- Identifies and documents all system constraints, implications and consequences of various proposed system changes.
- Educates end users on system designs and functionalities through creation of training materials and conducting demos.
- Evaluates the efficiency and effectiveness of application operations and troubleshooting problems.
Skills: Data Warehouse - Must Have - Snowflake, AWS Glue, Python, Pyspark.
Build and maintain data pipelines using AWS Glue and Python • Develop SQL-based transformations loading data into Snowflake data warehouse • Implement data quality checks and validation rules • Support production data pipeline monitoring and issue resolution • Document data workflows, processes, and technical specifications
Technical Execution :
Write clean, maintainable Python code following team standards • Execute ETL/ELT processes extracting data from multiple sources • Perform unit testing and participate in code reviews • Use Terraform for infrastructure provisioning under supervision • Troubleshoot and fix data pipeline issues with guidance
Technical Skills :
Proficiency in Python programming • Strong SQL skills and database knowledge • Familiarity with AWS services (Glue, S3) or willingness to learn • Good understanding of Snowflake or similar data warehouse platforms • Knowledge of version control (Git) and collaboration tools • Understanding of data modeling concepts