Lead Data Software Engineer

Accountabilities

• Design, build, and maintain scalable data pipelines that ensure reliable data flow across multiple systems.

• Develop and manage ingestion pipelines that collect and process data from diverse source systems.

• Write clean, maintainable, high-quality, and thoroughly tested code based on strong software engineering principles.

• Collaborate with cross-functional teams to understand business and data requirements and translate them into effective technical solutions.

• Monitor pipeline performance, identify potential issues proactively, and maintain system reliability.

• Participate in code reviews, uphold engineering quality standards, and share technical knowledge with team members.

• Investigate and resolve data quality issues to ensure the accuracy, consistency, and reliability of datasets.

• Contribute to the continuous improvement of data architecture, development practices, and engineering standards.

• Create and maintain technical documentation to support knowledge sharing, operational continuity, and onboarding.

• Participate in Agile ceremonies, including sprint planning and regular team meetings.

• Provide technical leadership and support the development team in delivering reliable data engineering solutions.

Requirements:

• At least 5 years of relevant professional experience.

• At least 1 year of experience leading and managing development teams.

• Strong experience with PySpark and Spark SQL for building and managing large-scale data processing workflows.

• Hands-on experience developing and managing Databricks Lakeflow pipelines and Jobs for data orchestration.

• Proficiency with Auto Loader and JDBC/CDC ingestion strategies, including experience integrating data from DB2 and SQL Server.

• Strong knowledge of Delta Lake and Unity Catalog for data storage, management, and governance.

• Experience implementing data quality controls and source reconciliation processes to ensure trustworthy data.

• Background writing unit tests with pytest to validate code quality and reliability.

• Working knowledge of Git and Databricks Asset Bundles for version control and deployment management.

• Upper-intermediate English proficiency or higher (B2).

• Experience participating in DataStage migration projects.

• Experience extracting and integrating data from Dynamics 365 or Dataverse.

• Databricks certification.

Benefits:

• Opportunity to work on international projects for leading global brands.

• Collaboration with highly skilled and diverse teams across different locations.

• Healthcare benefits.

• Employee financial programs.

• Paid time off and sick leave.

• Upskilling, reskilling, and professional certification opportunities.

• Unlimited access to LinkedIn Learning, including more than 22,000 courses.

• Global career development opportunities.

• Volunteer and community involvement initiatives.

• Employee Groups and opportunities to connect with colleagues.

• An award-winning workplace culture recognized by Glassdoor, Newsweek, and LinkedIn.