Big Data Engineer (PySpark & Scala)
Summary
The Big Data Engineer will design and build scalable data pipelines and streaming solutions using PySpark, Scala, and various big data technologies. The role focuses on optimizing distributed data platforms and processing large-scale datasets within an enterprise environment.
Role Summary
We are seeking a highly skilled Big Data Engineer with strong experience in PySpark and Scala to design, build, and optimize large-scale data processing systems. The candidate will work on distributed data platforms, enabling real-time and batch analytics solutions across enterprise data ecosystems.
The role involves building scalable data pipelines, data lakes, and streaming solutions leveraging modern big data technologies.
Key Responsibilities
Data Engineering & Development
Design and develop scalable data pipelines using PySpark and Scala
Data Platform & Architecture
Data Processing & Optimization
Streaming & Real-Time Data
Cloud & Modern Data Stack
Data Quality & Governance
Collaboration & Delivery
Required Skills
Core Technologies
Strong hands-on experience in:
PySpark & Apache Spark
Scala programming
Expertise in:
Spark SQL, DataFrames, RDDs
Distributed data processing frameworks
Big Data Ecosystem
Hadoop: HDFS, Hive, YARN
Data ingestion tools: Kafka, Sqoop
Data formats: Parquet, ORC, JSON, Avro
Programming & Query
Python / Scala
Advanced SQL (joins, aggregations, optimization)
Shell scripting / Unix basics
Streaming & Messaging
Kafka (must-have)
Event-driven architecture
Real-time data processing frameworks
DevOps & Tools
Git, CI/CD pipelines
Docker / Kubernetes (good to have)
Scheduling tools (Airflow / Control-M)
Cloud (One or more preferred)
AWS / Azure / GCP
Databricks / Snowflake exposure
ibu
itcan
Domain Experience (Preferred)
BFSI / Banking / Payments domain
Experience working with high-volume financial datasets
Knowledge of data compliance and regulatory reporting
This role requires a combination of:
Strong PySpark + Scala coding
Deep big data architecture knowledge
Hands-on experience with streaming + cloud data engineering