Data Engineer | Databricks Experience

Job Description

Job Summary:

Develops, maintains, and enhances business applications. Collaborates with stakeholders to validate user requirements, assess available technologies, and recommend solution options. Assesses objectives for assigned project phases and recommends technical strategies and tactics to achieve business needs.

Principal Responsibilities:

• Uses process design technology methodologies, programming languages and tools and solutions design techniques to develop applications to meet business specifications.
• Validates functional specifications, builds technical specifications and develops application documentation.
• Performs analysis, design, development and testing of applications to solve business requirements.
• Integrates new or enhanced application components.
• Develops conversion and system implementation plans.
• Supports change readiness initiatives as needed.
• Other duties as assigned.

Job Specifications:

• Design, build, and maintain data pipelines in Databricks using PySpark, SQL, and Delta Lake
• Develop reusable data transformations following medallion architecture patterns (bronze/silver/gold)
• Validate data accuracy, completeness, and consistency across datasets and pipeline stages
• Optimize jobs for performance, scalability, and cost efficiency
• Collaborate with business analysts and business stakeholders to translate requirements into data models and datasets for analytics
• Troubleshoot and resolve data pipeline issues, including root-cause analysis in production environments
• Document data pipelines, data lineage, and transformation logic
• Apply best practices for version control (GitHub), testing, and deployment
• Identify and implement improvements to data quality, pipeline reliability, and overall architecture

Work Experience:

• Typically 3+ years with bachelor's or equivalent.

Education and Certification(s):

• Bachelor's degree or equivalent experience from which comparable knowledge and job skills can be obtained.

Distinguishing Characteristics:

• Experience with transitioning Alteryx workflows to Databricks (Plus)
• Experience with PowerBI Visualizations