Remote .Net Data Analytics Engineer
Posted Updated
Description
Toptal is seeking a data engineer to design and optimize scalable data pipelines. You will implement Medallion architecture in Python, ensure cross-cloud compatibility with Parquet/Iceberg/Delta Lake, and embed analytics with DuckDB in ETL workflows.
You will also optimize PostgreSQL schemas, leverage SQLite for lightweight storage, and scale workloads using Databricks or Spark in distributed processing environments.
Pipeline Design: Build resilient data processing pipelines in Python following Medallion architecture (Bronze/Silver/Gold). Open Storage Management: Standardize data storage with Parquet, Iceberg, and Delta Lake for cross-cloud compatibility. Embedded Analytics: Use DuckDB integrated with Python for fast localized data processing within ETL workflows. Database Operations: Architect and optimize production PostgreSQL schemas; use SQLite for embedded storage when needed. Distributed Compute: Scale ETL and analytics with Databricks or Apache Spark. Advanced SQL with PostgreSQL expertise is required. Experience building analytical pipelines using Python and DuckDB. Proficiency with open storage formats (Parquet, Iceberg, Delta Lake). Proven track record designing data lakes with Medallion architecture. Familiarity with Databricks/PySpark for distributed processing.Skills
Sign in to read and join the discussion.