Data Engineer - Apache Spark

Job Title : Data Engineer

Key Responsibilities :

Data Pipeline Development :

- Design and develop scalable data pipelines to process and transform large volumes of structured and unstructured data.

Data Processing :

- Build and optimize data processing frameworks using Apache Spark.

Data Integration :

- Integrate data from multiple sources to create reliable and efficient data workflows.

Data Management :

- Ensure efficient storage, retrieval, and transformation of data for analytics and reporting.

System Performance :

- Monitor and optimize data pipelines for performance, scalability, and reliability.

Troubleshooting :

- Identify and resolve issues in data pipelines and processing systems to ensure data accuracy and availability.

Collaboration :

- Work closely with data scientists, backend engineers, and product teams to understand data requirements and deliver solutions.

Documentation :

- Maintain clear documentation of data pipelines, data models, and workflows.

Continuous Improvement :

- Stay updated with emerging technologies and best practices in big data and data engineering.

Required Skills :

- Strong proficiency with Apache Spark and distributed data processing.

- Experience designing and implementing scalable data pipelines.

- Knowledge of data integration techniques across heterogeneous sources.

- Ability to monitor, troubleshoot, and optimize data workflows for performance and reliability.

- Good communication skills for crossfunctional collaboration and documentation.