Data Engineer

Location: New York City, NY

About the Role

A Data Engineer designs, builds, and maintains the systems (data pipelines, databases, warehouses) that collect, process, and store large volumes of data, making it accessible and reliable for data scientists and analysts to derive insights. This role focuses on infrastructure, automation, performance, and security to ensure data quality and availability for business intelligence and machine learning.

Core Responsibilities

• Data Pipeline Development: Build and maintain ETL/ELT pipelines for data ingestion, transformation, and loading from various sources (APIs, databases).
• Data Architecture: Design and manage data models, schemas, and infrastructure (data lakes, warehouses like Snowflake, Redshift).
• Data Quality & Governance: Implement data validation, security (encryption, access control), and compliance measures.
• Performance Optimization: Monitor system performance, tune databases, and optimize data processing for efficiency.
• Collaboration: Work with analysts, scientists, and business stakeholders to understand data needs and deliver solutions.

Key Skills & Technologies

• Programming: Python, SQL
• Cloud Platforms: AWS, Azure, GCP
• Big Data Tools: Spark, Kafka, Hadoop
• Databases/Warehouses: Snowflake, Redshift, BigQuery, PostgreSQL, MongoDB
• DevOps/Tools: Docker, Kubernetes, Airflow (for orchestration)

What They Do

Data engineers are the plumbers of the data world, ensuring clean, structured data flows smoothly from raw sources to analytical tools, enabling data-driven decisions and powering machine learning models.