Software Engineer (Apache / Pyspark / Python) - Contract Role
Posted 1
view
Description
Key Responsibilities
- Design, develop, and optimize data workflows and pipelines using Apache Hadoop, Cloudera, and Spark.
- Develop microservice APIs using Python Django framework and utilized Redis as a data store
- Developed various datawarehouse applications on Mainframe with Teradata.
- Manage and optimize data storage in data warehouse solutions, including Cloudera and Hive.
- Develop a unified execution framework to oversee vCPU and memory configuration for YARN applications and logging processes.
- Implement and manage data processing workflows using tools like PySpark, Hive, MapReduce, and Impala.
- Build robust data ingestion frameworks with Kafka and other ETL tools, ensuring data quality and reliability.
- Work with structured and unstructured data using Python, Bash, and Perl scripting to analyze, process, and transform data.
- Manage MongoDB and ensure integration with other data storage and processing systems.
- Identify and address performance degradation issues such as Data Spill, Skew, and resource constraints on Spark Batch & Streaming applications in production environments.
- Use Apache Airflow to schedule, manage, and monitor ETL workflows.
- Work closely with data scientists, analysts, and other engineers to meet business requirements and drive data initiatives.
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
- 7+ years of experience in Big Data and data engineering roles.
- Proficiency with Cloudera, Hadoop, Hive, and MapReduce.
- Advanced knowledge of Python, Bash, and Perl.
- Strong skills in PySpark, Impala, and Kafka.
- Hands-on experience with data warehouse management and optimization.
- Experience with MongoDB and relational databases.
- Proficiency with Apache Airflow or similar orchestration tools.
- Knowledge of Apache, Linux environments, and other DevOps tools.
- Familiarity with cloud platforms such as AWS, Azure, or GCP.
- Experience with real-time data processing and streaming.
Skills
Sign in to read and join the discussion.