Big Data Engineer
Summary
Build and optimize big-data pipelines and analytics using Hadoop, PySpark, Scala, and Python to process large-scale datasets and integrate data from multiple sources.
Big Data Engineer
Job Requirement:
5 years of hands on work experience in Hadoop, Scala, Pyspark and Python.
Work with Hadoop Distributed File System (HDFS) to manage and store large datasets
Utilize tools like Hive, HBase, and Pig to process data in Hadoop
Set up and manage data pipelines to extract, transform, and load (ETL) large datasets
Develop and optimize PySpark jobs and Scala-based Spark applications to process and analyze large-scale data
Integrate data from various sources (relational databases, flat files, APIs, etc.) using Python and Spark.
Build data models and perform advanced analytics using PySpark, Python libraries and Scala.