Data Engineer
You will design, build, and maintain scalable data infrastructure and pipelines that ingest raw data from diverse sources, clean and validate it, and make it accessible for analysis and model training. You will craft efficient SQL and GraphQL queries, implement workflow orchestration, and manage the DevOps lifecycle of data tables and machine learning models. You will instrument monitoring, testing, and versioning to keep production data reliable and performant. You may also ingest and process blockchain data and integrate blockchain APIs into pipelines when required.
Responsibilities
- Design and implement data ingestion pipelines
- Develop data cleaning validation and preprocessing pipelines
- Create and optimize data access queries using SQL and GraphQL
- Manage lifecycle of data tables and machine learning models using DevOps practices
- Build scalable infrastructure for batch micro-batch and stream processing
- Integrate blockchain data sources and blockchain APIs into pipelines
- Implement workflow orchestration with tools like Airflow Dagster or DBT
- Ensure data quality monitoring testing and version control in production
Requirements
- 3+ years of professional data engineering experience
- Proficiency in one or more programming languages such as Java Scala or Python
- Advanced SQL experience with data warehouse systems such as Snowflake BigQuery or Databricks
- Experience with GraphQL SQL and NoSQL databases such as MySQL PostgreSQL Cassandra HBase Redis DynamoDB or Neo4j
- Experience with workflow orchestration tools such as Airflow Dagster or DBT
- Experience with cloud services such as AWS Google Cloud or Microsoft Azure
- Experience building scalable infrastructure for batch micro-batch and stream data processing
- Strong database management data modeling and DevOps knowledge for data and model management