DATA ENGINEER
Posted 8
views
Summary
Builds and maintains data storage solutions, ingestion pipelines, and transformation/cleansing processes, while monitoring platform stability and performance. Works mainly with Azure, Databricks, SQL, and Python (PySpark).
- 1. Designs and builds data storage solutions including database objects and data structures, data ingestion pipelines
- 2. Develops data transformation, cleansing, and preparation processes.
- 3. Perform job optimization/automation via code enhancement, configuration, or implementation of new tools/technology
- 4. Perform unit testing and system integration testing.
- 5. Maintain and monitor platform stability and performance of EDP. Execute updates, patches, and other activities required to maintain and
- enhance operations.
- 6. Ensures compliance with governance standards of the platform, security policies, and SDLC processes.
- 7. Collaborate with cross-functional teams to identify data needs and opportunities for improvement
- 8. Develop and maintain documentation related to data engineering and analytics processes and procedures
- 9. Perform Proof of Concept on new integration and analytic patterns and solutions
- 10.Contribute to best practices and standards for the data engineering team
Must-have:
- 1. Experience on Data projects is a MUST (at least 1 end to end project)
- 2. Experience in using Databricks and Azure (certifications preferred)
- 3. Experience in SQL, Python (Pyspark)
- 4. Knowledge in Data Warehouse and Data Mart concepts and query optimization techniques