Data Engineer
Posted Updated
Deskripsi Pekerjaan
Key Responsibilities
- Pipeline Development: Design, develop, and maintain robust ETL/ELT (Extract, Transform, Load) pipelines to move data from operational databases and external APIs into our data warehouse
- Data Modeling: Architect scalable data models (Star Schema, Snowflake, Data Vault) to ensure high performance and ease of use for end-users.
- Infrastructure Management: Manage cloud-based data environments (AWS, GCP, or Azure) and orchestrate workflows using tools like Apache Airflow or Prefect.
- Data Quality & Governance: Implement automated testing and monitoring to ensure data integrity, security, and compliance with privacy regulations (GDPR/CCPA).
- Collaboration: Partner with Data Scientists to prepare "feature stores" and with Business Intelligence (BI) Analysts to optimize query performance.
- Bachelor’s degree in Computer Science, Information Technology, or related field
- Proven experience as a Data Engineer or similar role
- Strong proficiency in SQL and database management (e.g., PostgreSQL, MySQL)
- Experience with programming languages such as Python, Java, or Scala
- Hands-on experience with ETL tools and data pipeline frameworks
- Familiarity with cloud platforms (e.g., AWS, GCP, or Azure)
- Understanding of data warehousing concepts and modeling (e.g., star/snowflake schema)
- Experience with big data technologies (e.g., Apache Spark, Hadoop)
- Knowledge of workflow orchestration tools (e.g., Airflow)
- Familiarity with containerization (Docker, Kubernetes)
- Experience with streaming data (e.g., Kafka)
- Understanding of data security and compliance standards