Data Engineer
Posted
Summary
Designs, builds, and maintains scalable data pipelines, ETL/ELT workflows, and data warehouses/lakes, ensuring data quality and supporting BI and reporting. Core stack includes SQL, Python/Java/Scala, Spark/Hadoop, Kafka, Airflow, and cloud platforms (AWS/Azure/GCP).
Job Responsibilities
- Design, develop, and maintain scalable data pipelines.
- Collect, process, transform, and manage large volumes of structured and unstructured data.
- Develop and maintain ETL/ELT workflows.
- Build and optimize data warehouses and data lakes.
- Integrate data from multiple sources and systems.
- Perform data cleaning, validation, and transformation.
- Monitor data pipeline performance and troubleshoot issues.
- Ensure data quality, accuracy, consistency, and reliability.
- Work with data analysts, data scientists, and business teams.
- Develop and optimize SQL queries for data processing.
- Implement data security and governance practices.
- Automate data integration and processing tasks.
- Maintain technical documentation for data workflows.
- Support business intelligence and reporting requirements.
Technical Skills
- Strong knowledge of SQL and database management.
- Programming knowledge in Python, Java, or Scala.
- Understanding of ETL and ELT processes.
- Knowledge of Data Warehousing concepts.
- Experience with databases such as MySQL, PostgreSQL, and SQL Server.
- Knowledge of Big Data technologies such as Apache Spark and Hadoop.
- Familiarity with Apache Kafka and Apache Airflow.
- Understanding of data modeling and database optimization.
- Knowledge of cloud platforms such as AWS, Azure, or Google Cloud.
- Understanding of data lakes and data lakehouse architecture.
- Knowledge of Git and version control systems.
- Basic understanding of Linux and shell scripting.
Preferred Skills
- Python / Java / Scala.
- SQL / MySQL / PostgreSQL.
- Apache Spark / PySpark.
- Hadoop.
- Apache Kafka.
- Apache Airflow.
- ETL / ELT Tools.
- AWS / Microsoft Azure / Google Cloud.
- Data Warehousing.
- Data Modeling.
- Snowflake / Databricks.
- Git / GitHub.
Soft Skills
- Strong analytical and problem-solving skills.
- Good communication and teamwork skills.
- Attention to detail.
- Ability to work with large and complex datasets.
- Critical thinking and logical reasoning.
- Time management and organizational skills.
- Ability to learn new technologies quickly.
Key Performance Indicators (KPIs)
- Data pipeline performance and reliability.
- Data processing accuracy and efficiency.
- Timely delivery of data engineering projects.
- Reduction in data processing errors.
- Data quality and consistency.
- ETL/ELT workflow optimization.
- Data availability and accessibility.
- Compliance with data security standards.
Skills
- Accessibility
- Airflow
- AWS
- Azure
- Bash
- Big Data
- Business Intelligence
- Cloud
- Data Engineering
- Data Lake
- Data Modeling
- Data Pipelines
- Data Quality
- Data Warehousing
- Databricks
- ELT
- ETL
- GCP
- Git
- GitHub
- Hadoop
- Java
- Kafka
- Lakehouse
- Linux
- MySQL
- PostgreSQL
- PySpark
- Python
- Scala
- Snowflake
- Spark
- SQL
- SQL Server
- Version Control
Sign in to read and join the discussion.