Data Engineer
Posted Updated
Description
Tkxel is seeking a Data Engineer for a short-term AI pipeline project to design and implement secure data integration and batch data pipelines. You will work with relational databases, APIs, and on-premises storage, building and validating data workflows for downstream AI and analytics applications.
The role emphasizes SQL proficiency, YAML/config-driven setup, and collaboration with security and IT teams to ensure compliant data handling and deployment within the client’s on-premises
Build and configure batch data pipelines using the platform’s low-code data pipeline builder, with support for event- or message-based integration where required. Securely connect to client data sources, including relational databases, APIs, file-based sources, on-premises storage, enterprise systems, and Kafka or similar messaging platforms. Write and validate SQL for data extraction, transformation, filtering, aggregation, lookup logic, and validation. Prepare, transform, clean, and validate data required by AI workflows, ML models, backend services, and frontend applications. Define and validate source-to-target mappings, data schemas, field mappings, data types, event/message schemas, and downstream input/output structures. Create and maintain YAML or configuration files for pipeline setup, data mappings, data source connections, event/message connections, and workflow inputs. Validate data ingestion, including source formats, schemas, load completeness, duplicate and null handling, retry/reprocessing behavior, event/message delivery, data freshness, and downstream availability. Perform data quality checks, including completeness, consistency, reconciliation, schema validation, and event/message validation. Define how relational, intermediate workflow, event/message, and output data should be stored, structured, and made available to downstream workflows. Collaborate with Backend, Frontend, DevOps, and client IT/security teams to ensure correct data formats, secure connectivity, and successful deployment. Support testing, troubleshooting, deployment, documentation, and handoff of data pipeline components within the client’s on-premises environment. Strong SQL and data transformation capabilities. Experience with secure enterprise data connectivity. Batch data ingestion pipelines with scheduling, retry, and validation. Familiarity with YAML/config-driven data pipelines setup. Experience with on-premises data integration and related security controls.Skills
Sign in to read and join the discussion.