Sr. Data Engineer
Posted 2
views
Summary
Senior data engineer (6-8 years) building and automating large-scale batch and real-time data pipelines. Core stack spans the Hadoop ecosystem (HDFS, Hive, Iceberg, Spark SQL), streaming with Kafka and Flink, orchestration via Airflow, and cloud-native deployment with Docker and Kubernetes.
Sr. Data Engineer – Big Data
- Experience: 6-8 Years
- Location: Bengaluru, KA
Key Skills
- Big Data Technologies
- Hadoop, HDFS, Hive
- Spark / Spark SQL
- SQL
- Real-Time and Batch Data Processing
Required Skills and Experience
- Extensive experience with Apache Kafka, Apache Flink, and other streaming technologies.
- Strong expertise in Hadoop ecosystem tools such as Hive, Iceberg, and Spark SQL.
- Good understanding of Apache Airflow for orchestrating complex data workflows and pipelines.
- Proven experience designing and implementing automated data pipelines and materialized views.
- Proficiency in Python, Unix/Linux, or similar technologies.
- Strong SQL skills, including experience with Oracle, SQL Server, or similar databases.
- Experience with graph databases such as Neo4j or JanusGraph and graph query languages.
- Experience with monitoring and CI/CD tools such as Prometheus, Grafana, Jenkins, and GitHub Actions.
- Strong engineering fundamentals, including data structures, algorithms, testing (JUnit/pytest), Git, and clean coding practices.
- Experience with cloud-native technologies, including Docker and Kubernetes (deployment, networking, scaling, and troubleshooting).
- 6+ years of relevant professional experience.
- Bachelor's degree in Computer Science, Engineering, or equivalent practical experience.