Data Engineer
Summary
Design and optimize data pipelines and scalable ETL/ELT processes using Databricks and Azure Data Factory. Day to day involves writing and tuning SQL, using PySpark/Python for large-scale processing, integrating sources into warehouses or data lakes, and monitoring pipelines for data quality and reliability.
Robotic Technologies Automation in Plano, TX is seeking a Data Engineer to design and optimize data pipelines using Databricks and Azure Data Factory. The role focuses on building scalable ETL/ELT processes and transforming data for analytics in a cloud environment.
You will write SQL, apply PySpark or Python, and integrate data from multiple sources into data warehouses or data lakes, while monitoring pipelines to ensure data quality and reliability.
Develop and maintain ETL/ELT pipelines using Azure Data Factory (ADF). Work with Databricks for data processing and transformation. Write and optimize SQL queries for data extraction and transformation. Work with PySpark or Python for large-scale data processing. Integrate data from multiple sources into data warehouses or data lakes. Monitor, troubleshoot, and optimize data pipelines. Collaborate with teams to ensure data quality and reliability. Experience with Databricks and Azure Data Factory (ADF). Strong understanding of ETL/ELT concepts. Good knowledge of SQL and data transformations. Basic experience with Python or PySpark. Understanding of Azure cloud services and data lake concepts.Skills
Sign in to read and join the discussion.