Data Engineer
Summary
Data Engineer designing and optimizing big data pipelines and distributed data systems using Hadoop, Spark, and Hive, building stream processing with Spark Streaming and Storm, tuning advanced SQL, and modeling dimensional data. Seeks 7–10 years of experience; Java/Python/Scala skills core, with Azure and ML as pluses.
About the Job
We are looking for an experienced Data Engineer with strong expertise in Big Data technologies, distributed data systems, and data processing frameworks. The ideal candidate should have hands-on experience with Hadoop, Spark, Hive, Java, Python, and advanced SQL, along with the ability to lead technical deliverables, optimize data processing workflows, and ensure data quality and performance.
Responsibilities
- Design, develop, and optimize Big Data solutions using Hadoop, Spark, Hive, and related technologies.
- Develop and tune complex Java, MapReduce, and Hive jobs to ensure efficient data processing.
- Lead and contribute to backend and distributed data systems while remaining hands-on with technical implementation.
- Analyze and validate source data to ensure data integrity and accuracy.
- Design and build dimensional data models to improve data accessibility, efficiency, and quality.
- Develop and maintain stream-processing solutions using Spark Streaming, Storm, and similar technologies.
- Write and optimize advanced SQL queries and perform database performance tuning.
- Support the deployment and auditing of data models and attributes to ensure accuracy.
- Manage technical deliverables and coordinate with relevant teams to prioritize projects according to business requirements.
- Collaborate with team members and stakeholders to resolve complex data, software, and technical issues.
Requirements
- 7–10 years of experience in Big Data Engineering or a similar role.
- In-depth knowledge of Hadoop, Cloudera, Spark, and related Big Data frameworks.
- Hands-on experience with Hadoop ecosystem technologies, including HDFS, MapReduce, Pig, Hive, HBase, Flume, and Sqoop.
- Strong programming and scripting skills in Java, Python, Scala, or similar languages.
- Good knowledge of Big Data querying tools such as Hive and Impala.
- Proficiency in advanced SQL, query optimization, and performance tuning.
- Experience with relational and NoSQL databases, particularly PostgreSQL and Cassandra.
- Experience with stream-processing technologies such as Spark Streaming and Storm.
- Knowledge of Linux and the ability to troubleshoot complex data, networking, and software issues.
- Experience with Azure cloud services is a plus.
- Familiarity with data science and machine learning technologies is an added advantage.
- Financial services domain knowledge is a plus.
- Strong planning, organizational, communication, and interpersonal skills.
- Ability to take ownership of deliverables, collaborate effectively, and support overall team objectives.
Skills
Sign in to read and join the discussion.