Lead / Staff Backend Engineer (Data Platform)

Responsibilities:

• Design, develop, and deliver end-to-end services for Different.ai 's innovative platform.
• Solve complex engineering challenges, including building high-throughput data pipelines for machine learning jobs handling petabyte-scale data.
• Lead and mentor a team of dynamic engineers, fostering a collaborative and innovative environment.
• Take ownership of critical architectural decisions.

Requirements:

• Experience: 7-12 years in software and data engineering.
• Solid understanding of data architecture principles and big data technologies/tools & Distributed data systems.
• Experience with workflow orchestration tools like Apache Airflow, Dagster, Composer, and Airbyte, etc.
• Experience with Real-time streaming (Kafka, Pulsar, Flink) & Query engines (Trino, Presto, Druid)
• Experience with Data lakehouse architectures (Iceberg, Delta Lake, Hudi) & Batch processing (PySpark/Spark, Hadoop)
• Advanced understanding of cloud platforms (AWS, GCP, Azure), modern data warehouses (Databricks, Snowflake, and Redshift), and databases (SQL family databases, MongoDB, and ElasticSearch).
• Deep expertise in Java. Python and GoLang, etc.
• Proficiency in Kubernetes, CI/CD tools, and deploying software solutions on-premises.