Data Engineer
Summary
Builds and optimizes Python-based data pipelines using Spark, PySpark, and HBase to process and transform large datasets for backend services and APIs.
Job Title: Data Engineer/ Python Developer
Experience: Minimum 9 years
Employment: Fulltime, 5 days onsite
Mandatory Skills: Python, Spark, Pyspark
Responsibilities
- Design, develop, and maintain software solutions using Python.
- Develop Python and pyspark programs for data analysis. Good working experience with Python to develop a custom framework for generating rules (similar to a rules engine).
- Develop Python code to gather data from HBase and design a solution to implement using Pyspark. Apache Spark DataFrames/RDDs were used to apply business transformations and utilize Hive Context objects to perform read/write operations.
- Develop back‑end services, APIs, and integrate databases.
- Optimize applications for performance, security, and maintainability.
- Troubleshoot and resolve software defects and issues.