Senior Data Engineer

Job Description

• Design, develop and deploy data tables, views and marts in data warehouses, operational data store, data lake and data virtualization.
• Perform data extraction, cleaning, transformation, and flow. Web scraping may also be part of the work scope in data extraction.
• Design, build, launch and maintain efficient and reliable large‑scale batch and real‑time data pipelines with data processing frameworks.
• Integrate and collate data silos in a manner that is both scalable and compliant.
• Collaborate with Project Manager, Data Architect, Business Analysts, Frontend Developers, Designers and Data Analyst to build scalable data‑driven products.
• Responsible for developing backend APIs and working on databases to support the applications.
• Work in an Agile Environment that practices Continuous Integration and Delivery.
• Work closely with fellow developers through pair programming and code review process.
• The team is expected to perform Data Warehousing tasks, mainly in AWS GCC, and manage APIs.

Qualifications

• Proficient in general data cleaning and transformation (e.g., SQL, pandas, R) to ensure data accuracy and consistency.
• Proficient in building ETL pipelines (e.g., SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS container tasks, EventBridge, AWS Glue, Spring).
• Proficient in database design and various databases (e.g., SQL, PostgreSQL, AWS S3, Athena, MongoDB, PostgreSQL/GIS, MySQL, SQLite, VoltDB, Cassandra).
• Experience in cloud technologies such as AWS, Azure, Google Cloud.
• Experience and passion for data engineering in a big‑data environment using cloud platforms such as AWS, Azure, Google Cloud.
• Experience with building production‑grade data pipelines, ETL/ELT data integration.
• Knowledge about system design, data structure and algorithms.
• Familiar with data modelling, data access, and data storage infrastructure like Data Mart, Data Lake, Data Virtualisation and Data Warehouse for efficient storage and retrieval.
• Familiar with REST API and web requests/protocols in general.
• Familiar with big‑data frameworks and tools (e.g., Hadoop, Spark, Kafka, RabbitMQ).
• Familiar with W3C Document Object Model and custom web scraping (e.g., BeautifulSoup, CasperJS, PhantomJS, Selenium, Node.js).
• Familiar with data governance policies, access control and security best practices.
• Comfortable in at least one scripting language (e.g., SQL, Python).
• Comfortable in both Windows and Linux development environments.
• Interest in being the bridge between engineering and analytics.