Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

QUESS ENGINEERING PTE. LTD.

New

Software Engineer (Apache / Pyspark / Python) - Contract Role

Posted 33 views

Summary

Contract role in Singapore for a big data engineer: design and optimize data pipelines with Hadoop, Cloudera, Spark/PySpark, Hive and Kafka; build Python/Django microservices with Redis; orchestrate ETL with Airflow; and tune Spark batch/streaming performance in production.

Key Responsibilities

  • Design, develop, and optimize data workflows and pipelines using Apache Hadoop, Cloudera, and Spark.
  • Develop microservice APIs using Python Django framework and utilized Redis as a data store
  • Developed various datawarehouse applications on Mainframe with Teradata.
  • Manage and optimize data storage in data warehouse solutions, including Cloudera and Hive.
  • Develop a unified execution framework to oversee vCPU and memory configuration for YARN applications and logging processes.
  • Implement and manage data processing workflows using tools like PySpark, Hive, MapReduce, and Impala.
  • Build robust data ingestion frameworks with Kafka and other ETL tools, ensuring data quality and reliability.
  • Work with structured and unstructured data using Python, Bash, and Perl scripting to analyze, process, and transform data.
  • Manage MongoDB and ensure integration with other data storage and processing systems.
  • Identify and address performance degradation issues such as Data Spill, Skew, and resource constraints on Spark Batch & Streaming applications in production environments.
  • Use Apache Airflow to schedule, manage, and monitor ETL workflows.
  • Work closely with data scientists, analysts, and other engineers to meet business requirements and drive data initiatives.

Requirements:

  • Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
  • 7+ years of experience in Big Data and data engineering roles.
  • Proficiency with Cloudera, Hadoop, Hive, and MapReduce.
  • Advanced knowledge of Python, Bash, and Perl.
  • Strong skills in PySpark, Impala, and Kafka.
  • Hands-on experience with data warehouse management and optimization.
  • Experience with MongoDB and relational databases.
  • Proficiency with Apache Airflow or similar orchestration tools.
  • Knowledge of Apache, Linux environments, and other DevOps tools.
  • Familiarity with cloud platforms such as AWS, Azure, or GCP.
  • Experience with real-time data processing and streaming.

Skills

Apply

See also

Software Engineering jobs by country — openings, pay and top skills →