Data Engineer
Description
Company: Celebal Technologies
Job Title: Data Engineer
Experience: 3–5 Years
Job Location: Bogotá, Colombia
Work Mode: Onsite / Hybrid
Job Type: Contract / Contract-to-Hire
Job Overview
Celebal Technologies is looking for a skilled Data Engineer with 3–5 years of experience in data engineering, strong hands-on expertise in Databricks, PySpark, and Medallion Architecture, and experience working with cloud platforms such as Microsoft Azure or AWS.
The ideal candidate will be responsible for developing scalable data pipelines, transforming and processing large datasets, and building reliable data solutions using modern cloud-based data engineering technologies.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
- Implement the Medallion Architecture (Bronze, Silver, and Gold layers) to organize, process, cleanse, and transform data.
- Develop ETL/ELT workflows for batch and incremental data processing.
- Work with Delta Lake tables and optimize data storage and processing performance.
- Integrate data from multiple sources, including databases, APIs, files, and cloud storage.
- Develop and optimize SQL queries for data transformation, validation, and analysis.
- Implement data quality checks, error handling, and monitoring to ensure data accuracy and reliability.
- Work with cloud services on Azure or AWS to build and manage data engineering solutions.
- Troubleshoot pipeline failures, resolve data processing issues, and improve overall performance.
- Collaborate with data architects, analysts, data scientists, and other engineering teams to deliver business requirements.
- Follow best practices for code quality, version control, testing, and deployment.
Required Skills and Qualifications
- 3–5 years of experience in Data Engineering or a related technical role.
- Strong hands-on experience with Databricks and PySpark.
- Practical experience implementing Medallion Architecture (Bronze, Silver, and Gold layers).
- Proficiency in Python and SQL.
- Experience developing and maintaining ETL/ELT pipelines and data transformation workflows.
- Experience working with Delta Lake and distributed data processing.
- Hands-on experience with at least one cloud platform: Microsoft Azure or AWS.
- Understanding of data warehousing concepts, data modeling, and data integration.
- Knowledge of data quality, pipeline monitoring, and performance optimization.
- Strong analytical, problem-solving, and communication skills.
Preferred Qualifications
- Experience with Azure Data Factory, Azure Data Lake Storage, or Azure Databricks.
- Experience with AWS services such as Amazon S3, AWS Glue, or related data services.
- Familiarity with Unity Catalog and data governance concepts.
- Knowledge of Git, CI/CD pipelines, and automated deployment practices.
- Experience with incremental data loads, partitioning, and Delta Lake optimization techniques.
- Familiarity with Agile development methodologies.
Work Location and Requirements
- Location: Bogotá, Colombia.
- Work Mode: Onsite / Hybrid.
- Candidates should be comfortable working from the Bogotá office according to project requirements.
Skills
Sign in to read and join the discussion.