Data Engineer

Job opportunity for Data Engineer at PT Sigma Global Teknologi in Kota Jakarta Barat, DKI Jakarta, Indonesia.

Kualifikasi

1. 1. Programming language
   • a. Phyton
   • b. Advance SQL (subquery, ROW_NUMBER, RANK, PARTITION BY, Indexing, Explain Plan, dan Query Tuning, Stored Procedures, Triggers, View, Materialize View, Constraints, indexing, query optimization)
2. 2. Database Systems
   • a. MySQL
   • b. PostgreSQL
   • c. Oracle
   • d. Snowflake
   • e. Unstructured Data
3. 3. Data modeling dan arsitektur
   • a. Distributed Data Storage Systems (partitioned data storage, sorting key, SerDes, data replication, caching and persistence)
   • b. Distributed Data Processing Systems (partitioning, predicate pushdown, sort by partition, maintaining size of shuffle blocks, window function, leveraging all cores and memory available in the cluster to improve concurrency)
   • c. Stream Data Processing (Real-time, Stream and Batch Processing)
4. 4. Tools
   • a. Data Pipelines and Automation (Airbyte)
   • b. Data ingestion in Message Queues
   • c. Data Wrangling operations – Pandas,numpy,re
   • d. Data Scraping – requests/BeautifulSoup/lxml/Scrapy
   • e. Interacting with External APIs and other Data Sources, Logging
   • f. Parallel processing Libraries – Dask, Multiprocessing
   • g. Data engineering tools (Apache Kafka, Apache Airflow)
   • h. Cloud Native Technologies (Serverless Computing, Virtual Instances, Containering – Docker, Orchestration -Kubernetes)
   • i. Data Visualization (Qlikview, Tableau)