Data Acquisition Platform Engineer
Posted Updated
Description
Astro Holdings Berhad in Jakarta is seeking a Python-focused software engineer to design and maintain scalable web crawlers, scrapers, and external API integrations across ecommerce and retail sources. You will build acquisition systems that stay reliable as websites and APIs evolve, handle sessions, pagination, tokens, and rate limits, and ensure robust testing and monitoring.
Collaborate with Pricing and Data teams to turn external-data needs into enduring platform capabilities using cloud
Build and maintain reliable web/app crawlers, scrapers, and external API integrations across e-commerce, marketplace, retail, and other external sources. Build acquisition systems that remain reliable as external websites, applications, APIs, and data contracts evolve. Handle real-world source complexity such as sessions, tokens, pagination, dynamic content, rate limits, retries, concurrency, and partial failures. Build resilient and maintainable acquisition systems with appropriate testing, idempotency, deduplication, retries, safe reruns, data-quality validation, monitoring, and alerting. Operate acquisition workloads using appropriate workflow orchestration, compute, data storage/warehousing, and cloud infrastructure. Continuously improve reliability, scalability, maintainability, and total cost of ownership, including evaluating build-vs-buy approaches for external data. Explore new technologies and AI-assisted development or automation to improve how external-data pipelines are built, operated, and maintained. Collaborate with Pricing, Data Warehouse Engineering, Data Analysts, and other stakeholders to turn external-data needs into sustainable platform capabilities. Strong Python and software engineering fundamentals. Hands-on experience building or operating data acquisition systems. Understanding of production reliability patterns such as retry, timeout, rate limiting, concurrency, and failure recovery. Solid knowledge of SQL, data storage/warehousing, schema evolution, workflow orchestration, containerized workloads, and cloud infrastructure fundamentals.Skills
Sign in to read and join the discussion.