Data Engineer
Summary
Designs and maintains scalable data pipelines, ETL workflows, and cloud-based data warehouses using Python, Spark, and cloud platforms like AWS/Azure.
🔑 Key Responsibilities
- Design, develop, and maintain scalable data pipelines.
- Build and optimize ETL/ELT workflows.
- Develop and manage data warehouses and data lakes.
- Integrate data from multiple sources.
- Optimize database performance and data quality.
- Collaborate with data scientists, analysts, and application teams.
- Implement data governance and security best practices.
- Monitor and troubleshoot data pipeline issues.
- Automate data processing workflows.
- Maintain technical documentation.
🛠 Technical Skills
- Python, SQL, Scala
- Apache Spark, Hadoop, Kafka
- ETL/ELT Development
- Data Warehousing
- Snowflake, Redshift, BigQuery
- AWS, Azure, Google Cloud Platform
- Airflow, Databricks
- SQL Server, PostgreSQL, MySQL
- Git, Docker, Kubernetes
✅ Mandatory Requirements
- 8+ years of Data Engineering experience.
- Strong SQL and Python programming skills.
- Experience with Spark and ETL frameworks.
- Hands-on experience with cloud data platforms.
- Knowledge of data warehousing concepts.
- Experience with workflow orchestration tools.
- Understanding of data modeling and optimization.
- Familiarity with Agile methodologies.
🎯 Core Competencies
- Data Engineering
- ETL/ELT Development
- Data Warehousing
- Cloud Platforms
- Big Data Technologies
- Database Management
- Performance Optimization
- Data Modeling