Data Engineer
Posted 48
views
Summary
Data engineer at Virtusa building end-to-end Azure data solutions: Azure Databricks with Unity Catalog for large-scale processing and governance, Python/PySpark and SQL ETL/ELT pipelines, Informatica IDMC (CDI) mappings, and Git-based CI/CD via Azure DevOps and GitHub Actions. Requires 5+ years of Azure data engineering experience.
- Expertise in Azure Databricks: Extensive experience with Azure Databricks for data engineering, data science, and analytics solutions.
- Unity Catalog: Proficiency in using Unity Catalog for comprehensive data governance, access control, and lineage tracking across multiple environments.
Advanced Python Skills
- Advanced proficiency in Python for data engineering tasks, including designing and implementing ETL/ELT pipelines and developing reusable, modular code for efficient data transformations.
- Familiarity with Python libraries like Pandas, NumPy, PySpark,
SQL Expertise
- Strong SQL skills to design complex queries and optimize database performance.
- Experience working with Azure databases like Azure SQL Database, Synapse Analytics, and Data Lake for querying, processing, and managing large datasets.
IDMC Data Integration
- Mapping design in Cloud Data Integration (CDI) — Source, Target, Expression, Aggregator, Joiner, Lookup, Filter, Router, Sorter, Rank transformations
- Mapping Task and Task Flow configuration and scheduling
- Parameterization (in/out parameters, parameter files) for reusable, environment-agnostic pipelines
- Working with Mapplets and reusable transformations
- Runtime environments — Secure Agent setup, agent groups, and troubleshooting agent connectivity
Git and CI/CD Expertise
- Version Control: Advanced experience with Git for managing code repositories, versioning, and collaborating in multi-developer environments.
- CI/CD Pipelines: Skilled in designing and implementing CI/CD pipelines using tools like Azure DevOps, GitHub Actions.
- In-depth knowledge of Azure Cloud architecture, including services like Azure Data Lake and Blob Storage.
- Hands-on experience in designing and deploying scalable, cloud-native data pipelines for large datasets.
Experience Summary:
- 5+ years of hands-on experience in delivering end-to-end data solutions using Azure technologies.
- Expertise in Azure Databricks with Unity Catalog for large-scale data processing and governance.
- Advanced skills in Python and SQL for efficient data transformation and analysis.
- Strong experience CI/CD pipelines, and working in Git-based environments for efficient deployment.
- Driving best practices in code quality, data architecture, and DevOps.