Senior Data Engineer

Location: Remote

Work hours: US (EST)

About the Role

Our client needs a Senior Data Engineer who can design and develop data pipelines to support system integrations and analytical platforms. They are looking for someone who can lead large-scale, enterprise-wide initiatives while collaborating with Product Owners and cross-functional technical teams. The ideal candidate should be able to deliver scalable, flexible, and high-quality solutions aligned with data architecture and governance strategies, leveraging modern cloud technologies such as Snowflake or Databricks.

Responsibilities include, but are not limited to:

• Design and develop data pipelines and integrations that are performant, scalable, and flexible.
• Work with project teams to deliver system integration pipelines, ensuring a high degree of reliability and resiliency.
• Work with business users and project teams to provide SME guidance, finalize pipeline requirements, and develop level of effort estimates.
• Develop automated testing and deployment scripts to support integrations and pipelines.
• Provide database support by coding utilities, responding to user questions, and resolving problems.
• Create and maintain documentation of data pipelines, including data flow and data lineage documentation.
• Be responsible for Tier 3 support and assist the Operations Team as required to provide a great customer experience for users.
• Be responsible for overall data quality as it relates to the Enterprise Data Warehouse and associated Data Marts. Provide data analysis as required to troubleshoot data issues.
• Assist with integration-related code reviews and mentor junior developers.
• Implement data governance and master data management principles as part of data pipeline development and delivery.

Qualifications

• Lakehouse Architecture: Hands-on experience building Lakehouse solutions on Databricks (Delta Lake, Unity Catalog) or Snowflake (Iceberg Tables, Horizon Catalog).
• Medallion Architecture: Proven ability to design Bronze/Silver/Gold layers for curated, analytics-ready data.
• Streaming or Batch Processing: Experience implementing Lambda or Kappa architectures using Databricks Structured Streaming / DLT or Snowflake Dynamic Tables / Snowpipe Streaming.
• Semantic Layer: Experience building and managing semantic models using Databricks AI/BI Genie or Unity Catalog Metrics or Snowflake Semantic Views / Cortex Analyst to enable consistent, governed metrics for BI and AI consumption.
• ETL/ELT Orchestration: Building pipelines with Airflow, Databricks Lakeflow, or Snowflake Openflow; familiarity with dbt is a plus.
• Programming: Strong PySpark or advanced SQL skills; Python for data engineering or automation.
• Data Modeling: Dimensional modeling, Data Vault, or schema design for analytical workloads.
• Performance Tuning: Cluster/warehouse sizing, partitioning, clustering keys, Z-ordering, or query optimization.
• Governance & Security: Working knowledge of Unity Catalog (Databricks) or Horizon Catalog (Snowflake) for lineage, access control, or data quality.
• CI/CD & DevOps: Git, Databricks Asset Bundles or Snowflake CLI/Schemachange, automated testing, or deployment pipelines.
• Cloud Ecosystem: AWS (S3, Glue, Kinesis), Azure, or GCP services supporting modern data platforms.
• AI/ML Enablement: Familiarity with Databricks Mosaic AI or Snowflake Cortex for GenAI or ML use cases.
• Collaboration: Strong communication skills to partner with Product Owners, Analysts, or cross-functional engineering teams.
• Education: Bachelor’s degree in Computer or Information Science or a related field, or an equivalent combination of education and experience.
• Experience: 5–8 years creating quality data pipelines and system integrations, with at least 3 years of experience in a cloud environment.

Must-Have Requirements

• Strong hands-on experience building data pipelines with Databricks, Apache Spark (PySpark), and Delta Lake.
• Advanced SQL and Python, with strong data modeling skills across dimensional and Lakehouse patterns.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience with legacy platform migrations, ETL modernization, or managing data hygiene when porting old systems.
• Strong problem-solving, collaboration, and communication skills, including mentoring junior engineers and explaining data concepts to non-technical stakeholders.
• Familiarity with Unity Catalog, data governance, access control, and PII handling.
• Experience with dbt or an equivalent transformation framework.
• Familiarity with secure coding standards and industry security best practices.

Nice-to-Have Requirements

• Experience with Infrastructure as Code (IaC) using Terraform and CI/CD using Azure DevOps.
• Experience working with relational databases, specifically PostgreSQL, and data persistence concepts