Big Data Engineer (PySpark & Scala)
Posted Updated 1
view
Role Summary
We are seeking a highly skilled Big Data Engineer with strong experience in PySpark and Scala to design, build, and optimize large-scale data processing systems. The candidate will work on distributed data platforms, enabling real-time and batch analytics solutions across enterprise data ecosystems. The role involves building scalable data pipelines, data lakes, and streaming solutions leveraging modern big data technologies. Key Responsibilities Data Engineering & Development Design and develop scalable data pipelines using PySpark and Scala Data Platform & Architecture Data Processing & Optimization Streaming & Real-Time Data Cloud & Modern Data Stack Data Quality & Governance Collaboration & Delivery Required Skills Core Technologies Strong hands-on experience in: PySpark & Apache Spark Scala programming Expertise in: Spark SQL, DataFrames, RDDs Distributed data processing frameworks Big Data Ecosystem Hadoop: HDFS, Hive, YARN Data ingestion tools: Kafka, Sqoop Data formats: Parquet, ORC, JSON, Avro Programming & Query Python / Scala Advanced SQL (joins, aggregations, optimization) Shell scripting / Unix basics Streaming & Messaging Kafka (must-have) Event-driven architecture Real-time data processing frameworks DevOps & Tools Git, CI/CD pipelines Docker / Kubernetes (good to have) Scheduling tools (Airflow / Control-M) Cloud (One or more preferred) AWS / Azure / GCP Databricks / Snowflake exposure ibu itcan Domain Experience (Preferred) BFSI / Banking / Payments domain Experience working with high-volume financial datasets Knowledge of data compliance and regulatory reporting This role requires a combination of: Strong PySpark + Scala coding Deep big data architecture knowledge Hands-on experience with streaming + cloud data engineering
We are seeking a highly skilled Big Data Engineer with strong experience in PySpark and Scala to design, build, and optimize large-scale data processing systems. The candidate will work on distributed data platforms, enabling real-time and batch analytics solutions across enterprise data ecosystems. The role involves building scalable data pipelines, data lakes, and streaming solutions leveraging modern big data technologies. Key Responsibilities Data Engineering & Development Design and develop scalable data pipelines using PySpark and Scala Data Platform & Architecture Data Processing & Optimization Streaming & Real-Time Data Cloud & Modern Data Stack Data Quality & Governance Collaboration & Delivery Required Skills Core Technologies Strong hands-on experience in: PySpark & Apache Spark Scala programming Expertise in: Spark SQL, DataFrames, RDDs Distributed data processing frameworks Big Data Ecosystem Hadoop: HDFS, Hive, YARN Data ingestion tools: Kafka, Sqoop Data formats: Parquet, ORC, JSON, Avro Programming & Query Python / Scala Advanced SQL (joins, aggregations, optimization) Shell scripting / Unix basics Streaming & Messaging Kafka (must-have) Event-driven architecture Real-time data processing frameworks DevOps & Tools Git, CI/CD pipelines Docker / Kubernetes (good to have) Scheduling tools (Airflow / Control-M) Cloud (One or more preferred) AWS / Azure / GCP Databricks / Snowflake exposure ibu itcan Domain Experience (Preferred) BFSI / Banking / Payments domain Experience working with high-volume financial datasets Knowledge of data compliance and regulatory reporting This role requires a combination of: Strong PySpark + Scala coding Deep big data architecture knowledge Hands-on experience with streaming + cloud data engineering