Data Engineer (PySpark, Apache Spark, Delta Lake, ETL, Oracle, DB2, AWS, TWS, Lambda, IAM)

Responsibilities

• Design, develop, and maintain scalable data pipelines for ingesting, transforming, validating, and delivering structured and semi-structured data.

• Develop data engineering solutions using Databricks, PySpark, and Apache Spark for large-volume data processing and transformation.

• Build and manage ETL/ELT workflows using modern data engineering platforms as well as enterprise ETL technologies.

• Develop data transformation and processing logic using SQL and Python, with a focus on performance, reliability, and data accuracy.

• Work with Delta Lake and Delta Live Tables to develop and maintain reliable data pipelines and curated data layers.

• Develop and optimize ETL workflows using IBM DataStage and Informatica PowerCenter where required.

• Work with relational databases including Oracle and IBM DB2 for data extraction, transformation, loading, querying, and performance optimization.

• Implement data integration solutions across cloud and enterprise data platforms.

• Develop and support data pipelines and associated services on AWS, including data storage, processing, monitoring, and related cloud services.

• Perform data validation, reconciliation, quality checks, and troubleshooting to ensure the accuracy and completeness of datasets.

• Optimize Spark jobs, SQL queries, ETL workflows, and data pipelines to improve processing efficiency and overall performance.

• Collaborate with technical and business teams to understand data requirements and translate them into scalable data solutions.

• Follow established development, deployment, documentation, data governance, and SDLC practices.

• Monitor production pipelines, investigate failures, and support timely resolution of data processing issues.
  

Requirements

• 5+ years of experience in data engineering, ETL development, data integration, or a related field.

• Strong hands-on experience with Databricks and data engineering workloads.

• Hands-on experience in PySpark and Apache Spark for distributed data processing.

• Solid experience developing ETL/ELT pipelines and data engineering solutions.

• Strong experience SQL skills, including complex queries, joins, aggregations, optimization, and data transformation.

• Hands on experience with AWS cloud services used for data engineering and data processing.

• Experience with Delta Lake, Delta Live Tables (DLT) is preferred.

• Hands-on experience with IBM DataStage and Informatica PowerCenter.

• Experience working with relational databases such as Oracle and IBM DB2.

• Hand on experience in Python for data processing, automation, or pipeline development.

• Experience with data modelling, data warehousing, data quality, and data integration concepts.

• Hands on experience in Lambda, Redshift, IAM, CloudWatch, Glue, EC2.

• Experience with production scheduling, pipeline monitoring, troubleshooting, and deployment processes.

• Experience in version control and CI/CD practices such as TWS, GIT, Jenkins etc.

• Ability to work effectively in an Agile/SDLC environment and collaborate with cross-functional teams.

• Data brick certification would be preferred.