Data Engineer
Posted 2
views
Summary
Designs, builds, and maintains ETL/ELT pipelines that move data from multiple sources into data warehouses and data lakes, including SQL transformations, Python automation, data quality checks, and pipeline monitoring. Core stack: SQL (PostgreSQL/MySQL/Oracle), Python, ETL tools like Airflow, Talend, or SSIS, plus Git and CI/CD practices.
- Design, develop, test, and maintain ETL/ELT pipelines from multiple data sources.
- Develop and optimize SQL queries, stored procedures, and data transformation logic.
- Build and maintain data integration workflows for data warehouses and data lakes.
- Perform data extraction, transformation, validation, and loading while ensuring data accuracy and consistency.
- Analyze source systems and create source-to-target mapping documentation.
- Monitor ETL jobs, troubleshoot failures, and optimize pipeline performance.
- Implement data quality checks, validation rules, and error-handling processes.
- Collaborate with Data Engineers, BI Developers, Analysts, and business stakeholders to understand data requirements.
- Document ETL processes, technical specifications, and data flows.
- Support deployment, maintenance, and continuous improvement of data integration solutions.
- Follow software development best practices, including version control, code reviews, and CI/CD where applicable.
- Design, develop, test, and maintain ETL/ELT pipelines from multiple data sources.
- Develop and optimize SQL queries, stored procedures, and data transformation logic.
- Build and maintain data integration workflows for data warehouses and data lakes.
- Perform data extraction, transformation, validation, and loading while ensuring data accuracy and consistency.
- Analyze source systems and create source-to-target mapping documentation.
- Monitor ETL jobs, troubleshoot failures, and optimize pipeline performance.
- Implement data quality checks, validation rules, and error-handling processes.
- Collaborate with Data Engineers, BI Developers, Analysts, and business stakeholders to understand data requirements.
- Document ETL processes, technical specifications, and data flows.
- Support deployment, maintenance, and continuous improvement of data integration solutions.
- Follow software development best practices, including version control, code reviews, and CI/CD where applicable.
Minimum Qualifications
- Bachelor's degree in Computer Science, Information Technology, Information Systems, Engineering, or a related field.
- Minimum 2-3 years of experience as an ETL Developer, Data Engineer, or similar role.
- Strong proficiency in SQL and relational databases (PostgreSQL, SQL Server, MySQL, Oracle, or similar).
- Experience developing ETL/ELT pipelines using tools such as SSIS, Talend, Informatica, Pentaho, DataStage, Airflow, or equivalent.
- Proficiency in Python for data processing and automation.
- Good understanding of data warehousing concepts, dimensional modeling (Star/Snowflake Schema), and data lake architecture.
- Experience working with APIs, JSON, XML, CSV, and other structured or semi-structured data sources.
- Familiarity with cloud platforms such as AWS, Azure, or Google Cloud Platform (GCP) is an advantage.
- Experience with Git or other version control systems.
- Strong analytical, troubleshooting, and problem-solving skills.
- Good communication skills and the ability to work collaboratively in an Agile environment.