L&T Technology - Data Engineer - ETL Tools
Description
Key Responsibilities :
- Design, develop, and maintain scalable data pipelines and ETL/ELT workflows.
- Build and optimize data processing solutions using Databricks and cloud-based data platforms.
- Develop data integration solutions using Azure Data Factory (ADF) and other ETL tools.
- Work with structured and unstructured data from multiple data sources.
- Develop and optimize data models, ensuring data quality, integrity, and consistency.
- Collaborate with data scientists, analysts, and business teams to understand data requirements and deliver scalable solutions.
- Monitor, troubleshoot, and optimize data pipelines for performance and reliability.
- Implement data governance, security, and best practices across the data platform.
- Support data migration, transformation, and modernization initiatives.
- Participate in code reviews, testing, deployment, and documentation activities.
Required Skills :
- Strong hands-on experience with Databricks.
- Experience with Azure Data Factory (ADF) or equivalent ETL tools.
- Proficiency in Azure and/or AWS cloud platforms.
- Good understanding of Hadoop ecosystem and distributed data processing.
- Strong experience in designing and developing ETL/ELT pipelines.
- Proficiency in Python, SQL, or Scala for data engineering.
- Experience with relational and NoSQL databases.
- Knowledge of data warehousing concepts and dimensional data modeling.
- Familiarity with version control tools such as Git.
- Strong analytical, debugging, and performance optimization skills.
Preferred Skills:
- Experience with Apache Spark and PySpark.
- Knowledge of Delta Lake, Apache Kafka, or streaming data pipelines.
- Exposure to CI/CD pipelines using Azure DevOps, Jenkins, or similar tools.
- Familiarity with containerization technologies such as Docker and Kubernetes.
- Experience with data governance and cloud security best practices.
Educational Qualification:
- B.E./B.Tech in Computer Science, Information Technology, or a related discipline.
- MCA or M.Sc. in Computer Science, Information Technology, or a related field is also acceptable.
Skills
- AWS
- Azure
- Azure Data Factory
- Azure DevOps
- CI/CD
- Cloud
- Cloud Security
- Containerization
- Data Engineering
- Data Governance
- Data Modeling
- Data Pipelines
- Data Quality
- Data Warehousing
- Databricks
- Delta Lake
- DevOps
- Docker
- ELT
- ETL
- Git
- Hadoop
- Jenkins
- Kafka
- Kubernetes
- NoSQL
- PySpark
- Python
- Scala
- Spark
- SQL
- Version Control
Sign in to read and join the discussion.