Data Engineer

Job Summary

We are seeking a skilled Data Engineer to design, develop, and maintain scalable data pipelines and modern data platforms. The ideal candidate will have hands-on experience in data engineering, ETL/ELT processes, cloud technologies, big data frameworks, and data warehousing solutions. The role involves collaborating with data scientists, analysts, and business stakeholders to deliver reliable and high-quality data solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and data integration solutions.
• Build and optimize ETL/ELT workflows for processing large-scale datasets.
• Develop and manage enterprise data warehouses, data lakes, and lakehouse architectures.
• Integrate data from multiple structured and unstructured sources.
• Ensure data quality, governance, security, and compliance standards.
• Optimize data storage, retrieval, and processing performance.
• Collaborate with Data Scientists, BI Developers, Product Owners, and business teams.
• Monitor and troubleshoot data pipelines and production data issues.
• Implement data validation, lineage, and monitoring frameworks.
• Support real-time and batch data processing requirements.

Required Skills

Core Technical Skills

• Strong proficiency in SQL and database design principles.
• Hands-on experience with Python, PySpark, or Scala.
• Experience building ETL/ELT pipelines and data integration solutions.
• Strong understanding of Data Modeling (Star Schema, Snowflake Schema, Dimensional Modeling).
• Experience with Data Warehousing concepts and best practices.
• Knowledge of batch and real-time data processing frameworks.

Big Data Technologies

• Apache Spark
• Hadoop Ecosystem
• Databricks
• Apache Kafka
• Apache Airflow

Cloud Platforms

• Microsoft Azure (ADF, ADLS, Synapse, Fabric)
• AWS (S3, Glue, Redshift, EMR)
• Google Cloud Platform (BigQuery, Dataflow)

Databases

• SQL Server
• PostgreSQL
• Oracle
• MySQL
• MongoDB

Preferred Skills

• Experience with Microsoft Fabric and modern Data Lakehouse architecture.
• Hands-on experience using Azure Data Factory (ADF), Synapse Analytics, and Databricks.
• Experience with streaming technologies such as Kafka and Spark Streaming.
• Knowledge of data governance and cataloging tools.
• Experience with CI/CD implementation for data pipelines.
• Familiarity with Infrastructure as Code (Terraform).

Tools & Technologies

• Python
• SQL
• PySpark
• Databricks
• Apache Spark
• Kafka
• Airflow
• Azure Data Factory
• Azure Synapse Analytics
• Microsoft Fabric
• Git
• Azure DevOps
• Jenkins

Educational Qualification

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field.

Good-to-Have

• Experience working in Retail, E-commerce, Banking, or Enterprise Data Platforms.
• Understanding of Machine Learning data pipelines.
• Knowledge of data governance, data security, and compliance frameworks.
• Strong analytical and problem-solving skills.
• Experience working in Agile/Scrum environments.