Data Engineer
Summary
Design and maintain Python-based ETL pipelines to ingest public economic/financial API data into Azure Blob and SQL databases for analytics and reporting.
Location: Jakarta
Position Type: Full-time (in office)
Reports to: CTO
About the Role
We’re looking for a Data Engineer to design, implement, and maintain reliable data pipelines that collect and process data from various public APIs into a centralised blob storage and database environment. You’ll play a key role in ensuring data is ingested efficiently, transformed cleanly, and made accessible for downstream analysis and reporting.
This role involves working with Python-based ETL pipelines, workflow orchestration (Apache Airflow or equivalent), and cloud storage solutions (e.g., Azure Blob Storage).
Key Responsibilities
- Design and implement robust ETL/ELT pipelines to extract data from multiple public APIs and load into blob storage and relational databases.
- Build and manage scheduled workflows using tools such as Apache Airflow (or similar).
- Handle complex API integrations, authentication flows (OAuth2, API keys), pagination, and rate limiting.
- Structure data in blob storage (e.g., Azure Blob) for efficient downstream access.
- Transform and load data into structured SQL-based databases for analytics and access by Web applications
- Implement monitoring, logging, and validation to ensure data integrity and pipeline reliability.
- Work closely with data scientists, and software developers to ensure data availability and usability.
- Maintain clear documentation for data sources, transformations, and orchestration logic.
Qualifications
- Bachelor’s degree in Computer Science, Data Engineering, or a related field (or equivalent experience).
- 3+ years of experience building data pipelines in a production environment.
- Strong understanding of ETL/ELT architecture, API data ingestion, and data modelling principles.
- Experience with Azure
Technical Skills:
- Languages: Python (preferred), SQL (Microsoft T-SQL)
- Orchestration: Apache Airflow or similar
- Familiarity with version control (Git) and CI/CD pipelines
Nice to Have:
- Experience integrating data from open economic and financial APIs (e.g., World Bank, IMF, OECD, UN Data, FRED)
- Familiarity with large structured and semi-structured datasets (JSON, CSV, Parquet) and associated performance considerations
- Experience building incremental ingestion or change data capture (CDC) pipelines
Soft Skills
- Strong analytical and problem-solving mindset
- Clear communication of technical concepts to non-technical stakeholders
- Attention to detail and data accuracy
- Proactive approach to monitoring
What We Offer
- Modern, central office near to public transport and key amenities
How to Apply
If you're interested in this role, please send your CV and Cover Letter to ame@amegroup.com .