NC 2 Backend Engineer

Job Description for Data Engineer / Data Specialist

1. Position Summary

2. Key Responsibilities

• Prepare and transform large volumes of data (millions of records) for analytics and reporting purposes.
• Load and monitor data into databases, primarily Postgres and Spark-based environments.
• Understand, maintain, and troubleshoot data pipelines, including scheduled data ingestion and job monitoring (e.g., handling job failures).
• Collaborate and effectively communicating with business teams to gather and understand data requirements, effectively translating requirements into data solutions as a forward deployed engineer.
• Coordinate with visualization and analytics teams to ensure data availability and integrity for reporting tools.
• Implement and enforce best practices for data preparation and curation in line with a curated data layer approach.

3. Mandatory Skills and Experience

• Strong experience working with large datasets (e.g., handling tables with millions of records).
• In-depth knowledge of data warehousing concepts, specifically experience with Postgres databases.
• Practical experience with Apache Spark (including PySpark), focusing on data processing within Spark clusters.
• Familiarity with modern data ingestion concepts, including scheduling, automated job monitoring, and troubleshooting failures.
• Experience working with data preparation layers rather than solely focusing on traditional ETL coding.
• Data preparation for consumption by data visualization tools like Tableau, with focus on data back-end support rather than report creation itself.

• Strong understanding of the differences and nuances between data platforms such as Spark and traditional databases.
• Familiarity with the business implications of fiscal and calendar years in analytics and reporting.
• Ability to communicate technical concepts clearly and interact confidently with business stakeholders.

• Excellent communication skills for interacting closely with business teams and stakeholders for requirement gathering and fulfillment.
• Proactive attitude towards vetting data quality and data pipeline reliability.
• Detail-oriented with the ability to explain and document technical details effectively.
• Ability to work in a hybrid team model and collaborate with cross-functional teams.

4. Preferred Qualifications

• Prior experience in a retail or large enterprise environment dealing with large-scale data.
• Experience with workload orchestration or scheduling tools is advantageous.
• Knowledge of additional data technologies such as Iceberg tables or Trino can be an added benefit but is not mandatory.

5. Additional Notes

• Candidates should be able to clearly articulate their role and contribution in previous data projects, focusing on the data volumes handled and technical responsibilities.
• Screening will emphasize practical experience with data preparation and loading rather than just ETL programming.
• Understanding of evolving data engineering roles, including forward-deployed engineering approaches, is a plus.