Senior Data Engineer
Description
Senior Data Engineer
Location: Beverly Hills, CA (in person)
ABOUT THE ROLE
We are hiring a Senior Data Engineer with 5+ years of experience to design, build, and operate a production data platform end to end. You will own ingestion, transformation, orchestration, and data quality on a modern cloud warehouse and lakehouse stack: Amazon Redshift or Databricks, Apache Airflow, dbt, and Apache Spark.
This is a hands-on role for someone who has run these systems in production and is accountable for their performance, cost, and reliability.
WHAT YOU'LL DO
- Own data models and pipelines on Amazon Redshift or Databricks, including performance tuning, cost management, and workload isolation.
- Author and operate Apache Airflow DAGs at scale, managing dependencies, retries, SLAs, and observability.
- Build and maintain dbt transformations: model design, testing, macros, documentation, and CI/CD.
- Develop and run ingestion pipelines using tools such as Fivetran, Airbyte, AWS DMS, Kafka/Kinesis, or custom CDC and API ingestion frameworks.
- Manage lakehouse tables in open table formats (Apache Iceberg, Delta Lake, or Hudi), covering partitioning, schema evolution, compaction, and time travel.
- Write large-scale batch and streaming transformations in PySpark or Spark SQL.
- Write advanced, well-optimized SQL and production-grade Python across the platform.
- Put data governance, quality checks, and observability in place so issues are caught before they reach data consumers.
REQUIRED QUALIFICATIONS
Candidates should demonstrate 5+ years of data engineering experience, with depth in:
- Cloud data warehouse / lakehouse: Amazon Redshift or Databricks, with real production ownership of modeling, performance tuning, cost management, and workload isolation.
- Python: strong, production-level proficiency.
- Apache Airflow: authoring and operating DAGs at scale, including dependency management, retries, SLAs, and observability.
- dbt: model design, testing, macros, documentation, and CI/CD for transformations.
- Data movement / ingestion tooling: for example Fivetran, Airbyte, AWS DMS, Kafka/Kinesis, or custom CDC and API ingestion frameworks.
- Open table formats: Apache Iceberg, Delta Lake, or Hudi, including partitioning, schema evolution, compaction, and time travel.
- Apache Spark: PySpark or Spark SQL for large-scale batch and streaming transformation.
- SQL: advanced, including window functions and query optimization.
- Data governance, quality, and observability: hands-on experience with tooling such as Great Expectations, Monte Carlo, or Elementary.
NICE TO HAVE
- AWS ecosystem depth: S3, Glue, Lambda, IAM, and EMR.
- Experience with streaming or near-real-time pipelines.
- Infrastructure-as-code (Terraform), Docker, and CI/CD (GitHub Actions).
Skills
Sign in to read and join the discussion.