Data Engineer (0665)
Posted Updated
Summary
Data Engineer (4+ years) building data pipelines and data assets on AWS; day to day involves PySpark/Spark SQL code optimization, legacy code refactoring, TDD, and data migration work including Redshift migration. Core stack: Python, Boto3, Databricks, S3, EC2, Lambda, Redshift, CloudFormation, Git.
Job Description:
Keywords: Data Engineer, Databricks, SQL + (Apache Spark or PySpark), Data migration project, code optimisation using spark/pyspark, Redshift migration
Data Engineer with a minimum of 4+ years of experience of data engineering experience. The role will require deep knowledge of data engineering techniques to create data pipelines and build data assets.
- At least 4+ years of Strong hands-on programming experience with Pyspark / Python / Boto3 including Python Frameworks, libraries according to python best practices.
- Strong experience in code optimization using spark SQL and pyspark.
- Understanding of Code versioning, Git repository, JFrog Artifactory.
- AWS Architecture knowledge specially on S3, EC2, Lambda, Redshift, CloudFormation etc and able to explain benefits of each
- Code Refactorization of Legacy Codebase: Clean, modernize, improve readability and maintainability.
- Unit Tests/TDD: Write tests before code, ensure functionality, catch bugs early.
- Fixing Difficult Bugs: Debug complex code, isolate issues, resolve performance, concurrency, or logic flaws.