Data Engineer Support
Summary
Builds and maintains data pipelines, ETL/ELT processes, and SQL scripts to extract, transform, and load data for business use. Focuses on Apache Spark, IBM InfoSphere, Informatica, and database technologies.
- Design, develop, and maintain ETL/ELT processes for data extraction, transformation, and loading.
- Build, optimize, and manage data pipelines to ensure efficient and reliable data processing.
- Develop and maintain SQL scripts, stored procedures, and database objects.
- Design and implement data models, data warehouses, and data architectures to support business needs.
- Perform data integration across multiple platforms and heterogeneous data sources.
- Utilize Apache Spark (PySpark, SparkR, or Scala) for large-scale data processing.
- Develop automation solutions for data workflows and integration processes.
- Collaborate with business users, analysts, and technical teams to gather requirements and deliver data solutions.
- Monitor, troubleshoot, and optimize ETL jobs and data pipelines to ensure performance and reliability.
- Ensure data quality, integrity, security, and compliance with organizational standards.
- Document technical designs, workflows, and operational procedures.
- Minimum 1 year of experience in Data Engineering or a related field, or equivalent practical experience.
- Diploma or Bachelor's degree in Informatics Engineering, Computer Science, Information Systems, or a related field.
- Strong experience with SQL and database development.
- Experience using SQL Server Integration Services (SSIS).
- Hands-on experience with IBM InfoSphere products, including DataStage, Information Analyzer (IA), Business Process Manager (BPM), Rational Application Developer (RAD/Workbench), Performance Management Edition (PME), and DB2.
- Experience with Informatica Data Integration / Informatica PowerCenter or equivalent Informatica solutions.
- Experience using Apache Spark for data processing with PySpark, SparkR, or Scala.
- Proficiency in Python and/or R programming.
- Experience in designing and implementing ETL/ELT processes.
- Solid understanding of data modeling, data architecture, and data warehousing concepts.
- Knowledge of data integration, information management, data processing, and workflow automation across multiple platforms.
- Strong understanding of various database technologies, including relational and non-relational databases.
- Strong analytical, problem-solving, and communication skills.
- Ability to work independently as well as collaboratively in a team environment.