AI HUB Data Engineer

Data engineering is the practice of making the appropriate data available to various data consumers (including data scientists, data and business analysts, citizen integrators, and line-of-business users). It is a discipline that involves collaboration across business and IT units.

Role at a glance:

• Main focus: Data pipelines, data architecture, integration, quality, reliability, and scalable data products.

• Typical outputs: ETL/ELT pipelines, curated datasets, data models, reusable data products, monitoring logic, and technical documentation.

• Key interfaces: Data scientists, Palantir/Foundry experts, IT, cybersecurity, data governance, business analysts, and Quality stakeholders.

In addition to creating and maintaining an optimal pipeline architecture, typical duties and responsibilities for a Data Engineer position may include:

Key Responsibilities:

· Assembling large, complex sets of data that meet non-functional and functional business requirements

· Design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.

· Develop robust ETL (Extract, Transform, Load) process to integrate data from various sources.

· Identifying, designing and implementing internal process improvements including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes

· Building required infrastructure for optimal extraction, transformation and loading of data from various data sources using AWS, Azure, Databricks, Palantir and SQL technologies

· Building scalable tables to provide actionable insight into key business performance metrics including operational efficiency and customer acquisition

· Working with stakeholders including the Data Product teams to support their data infrastructure needs while assisting with data-related technical issues

· Design and maintain data models, schemas, and database structures to support analytical and operational use cases.

· Optimize data storage and retrieval mechanisms for performance and scalability.