Data Engineer
Data Engineers design and build data systems and pipelines. Responsibilities include developing data processing workflows, optimizing data storage, and ensuring data accuracy. You will collaborate with data scientists and analysts to meet data requirements and resolve data issues. Strong experience in data engineering and problem-solving skills are required.
Design and implement data pipelines, Optimize data processing and storage, Ensure data solutions meet performance standards, Provide technical support, Collaborate with stakeholders.
8+ years in data engineering/ETL roles, with at least 4+ years in Azure cloud ETL leadership. Azure certifications (e.g., Azure Analytics Specialty, Solutions
Required Qualifications:
- Azure Data Sources: Azure Data Lake Storage (ADLS), Blob Storage, Azure SQL Database, Synapse Analytics. External Sources: APIs, on-prem databases, flat files (CSV, Parquet, JSON).
- Tools: Azure Data Factory (ADF) for orchestration, Databricks connectors.
- Apache Spark: Strong knowledge of Spark (PySpark, Spark SQL) for distributed processing.
- Data Cleaning & Normalization: Handling nulls, duplicates, schema evolution.
- Performance Optimization: Partitioning, caching, broadcast joins.
- Delta Lake: Implementing ACID transactions, time travel, and schema enforcement.
- Azure Data Factory (ADF): Building pipelines to orchestrate Databricks notebooks.
- Azure Key Vault: Secure credential management.
- Azure Monitor & Logging: For ETL job monitoring and alerting.
- Networking & Security: VNET integration, private endpoints.