Azure Databricks Data Engineer
Summary
Data engineer building and optimizing data pipelines in Azure Databricks: ingesting, transforming, and monitoring data with SQL, Python, and Azure services like Data Factory, Synapse, and SQL Database, while working directly with business users to translate requirements into technical designs.
Job Description
Must-Have Technical and Functional Skills
● Strong working knowledge of relevant Azure services.
● Strong knowledge of data warehouse concepts.
● Strong hands-on proficiency in working with data using SQL.
● Ability to write complex queries aligned with business requirements.
● Experience developing data pipelines in Databricks.
● Prior experience with a programming language, preferably Python.
● Hands-on experience using version control systems such as Git.
● Experience creating project plans, technical design documents, and training documents,
and presenting solutions to end users.
● Strong verbal and written communication skills, with the ability to express complex
technical concepts in business terms and translate complex business rules into technical
designs.
Roles and Responsibilities
● Strong knowledge of data warehousing and RDBMS platforms such as SQL Server,
Teradata, and Oracle.
● Hands-on experience with Python, Java, Airflow, and related technologies.
● Experience building and optimizing complex data pipelines in Azure or an equivalent cloud
platform.
● Strong working knowledge of Azure components and services such as Storage Accounts,
Azure SQL Database, Azure Data Factory, Microsoft Entra ID (formerly Azure Active
Directory), and Azure Synapse Analytics.
● Experience with Azure SQL Database deployment and administration is desirable.
● Knowledge of Azure networking services.
● Knowledge of Azure infrastructure creation using ARM templates.
● Proficiency with GitLab and Azure DevOps.
● Implement data pipelines to automate the ingestion, transformation, and augmentation of
data sources, and apply best practices for pipeline operations.
● Identify downstream implications of data loads and migrations, including data quality and
regulatory considerations.
● Monitor workloads, ensure optimal utilization of SQL scripts, document database
requirements, and analyze data anomalies.
● Work directly with business users and analysts to understand business requirements.
Generic Managerial Skills
● Ability to collaborate effectively across multiple teams.