Senior Data Engineer
Summary
Senior Data Engineer to design, build, and optimize big data solutions using Hadoop, Spark, Hive, and Spark Streaming, while leading data engineering teams and owning end-to-end technical delivery. Requires 10+ years of experience with strong Java, Python, Scala, and advanced SQL skills.
About the Job
We are looking for an experienced Senior Data Engineer with strong expertise in Big Data technologies, distributed data systems, and data engineering solutions. The ideal candidate will have hands-on experience with Hadoop, Spark, Java, Python, and advanced SQL, along with proven leadership skills to manage data engineering teams, oversee technical deliverables, and ensure data quality, performance, and reliability.
Responsibilities
- Design, develop, and optimize Big Data solutions using Hadoop, Spark, and related technologies.
- Develop and tune complex Java, MapReduce, and Hive jobs for efficient data processing.
- Lead backend and distributed data systems teams while remaining hands-on with technical implementation.
- Manage data engineering teams, including full-time employees and third-party resources.
- Coordinate with business intelligence teams and vendor partners to prioritize projects based on business and customer requirements.
- Analyze and validate source data to ensure data integrity and accuracy.
- Lead the deployment and auditing of data models and attributes to maintain accuracy.
- Design and build dimensional data models to improve data accessibility, efficiency, and quality.
- Develop and maintain stream-processing solutions using Spark Streaming, Storm, and similar technologies.
- Write advanced SQL queries and perform query optimization and performance tuning.
- Take ownership of Big Data team deliverables and ensure timely delivery.
- Collaborate with stakeholders and team members to resolve complex data, software, and networking issues.
Requirements
- 10+ years of experience in Big Data Engineering or a similar role.
- In-depth knowledge of Hadoop (Cloudera), Spark, and related Big Data frameworks.
- Strong understanding of Hadoop ecosystem technologies, including HDFS, MapReduce, Pig, Hive, HBase, Flume, and Sqoop.
- Proficiency in programming and scripting languages, including Java, Python, and Scala; knowledge of C++, Ruby, PHP, or R is an advantage.
- Good knowledge of Big Data querying tools such as Pig, Hive, and Impala.
- Strong expertise in advanced SQL, query optimization, and performance tuning.
- Experience with relational and NoSQL databases, particularly PostgreSQL and Cassandra.
- Hands-on experience with stream-processing technologies such as Spark Streaming and Storm.
- Experience in managing technical teams and owning end-to-end delivery of data engineering projects.
- Strong problem-solving, planning, organizational, and interpersonal communication skills.
- Ability to collaborate effectively, support team members, and maintain productive working relationships.
- Experience with Azure cloud services is a plus.
- Knowledge of data science and machine learning technologies is an added advantage.
- Financial services domain knowledge is a plus.
Skills
Sign in to read and join the discussion.