Data Engineer

• Jobseeker Video Testimonials
  
• Employee Glassdoor Reviews
  

We are an IT Solutions Integrator/Consulting Firm helping our clients hire the right professional for an exciting long-term project. Here are a few details.

Requirements

Job Description – Senior Data Engineer


Role Summary

We are looking for an experienced and hands-on Data Engineer to design, develop, and maintain scalable data pipelines and analytics solutions. The successful candidate will have strong programming capabilities in Python, advanced SQL expertise, and practical experience working with AWS-based data engineering services.

The role involves integrating data from multiple structured and unstructured sources, transforming and organizing large datasets, implementing data-quality controls, and preparing reliable data layers for analytics, reporting, and business intelligence solutions.

Key Responsibilities

· Design, develop, test, and maintain scalable batch and event-driven data pipelines.

· Build end-to-end ETL/ELT workflows using Python, SQL, and AWS data services.

· Develop reusable Python components for data ingestion, transformation, validation, reconciliation, and automation.

· Write and optimize complex SQL queries, stored procedures, views, functions, and data-transformation logic.

· Integrate data from databases, APIs, enterprise applications, flat files, and cloud-based file systems.

· Work with file-based datasets in formats such as CSV, JSON, XML, Parquet, and Excel.

· Query and analyse data stored in databases.

· Work with relational databases hosted on Amazon RDS, Azure Synapse, including Microsoft SQL Server or PostgreSQL.

· Implement incremental data loads, change-data-capture patterns, error-handling mechanisms, restartability, and audit controls.

· Perform data validation, reconciliation, profiling, cleansing, and quality monitoring.

· Apply appropriate security controls, including IAM-based access, encryption, logging, and secure handling of sensitive data.

· Collaborate with Power BI or Tableau developers to deliver analytics-ready datasets and semantic models.

Mandatory Technical Skills

Programming

· Strong hands-on programming experience in Python.

· Good understanding of object-oriented programming, modular development, exception handling, logging, and reusable coding practices.

· Prior exposure to another programming language, preferably Java, is desirable.

· Experience using Python libraries for data processing and database integration, such as Pandas, PySpark, Boto3, SQLAlchemy, or equivalent libraries.

SQL and Databases

· Strong proficiency in writing and optimizing advanced SQL queries.

· Sound knowledge of relational database concepts, normalization, denormalization, indexing, and database performance.

· Hands-on experience with Complex joins and subqueries, Window and analytical functions, Stored procedures, functions, and views, Query-performance optimization & Dimensional and relational data modelling

AWS Data Engineering

· Hands-on experience with several of the following services:

• Amazon S3
  
• AWS Glue
  
• AWS Lambda
  
• AWS Step Functions
  
• Amazon Redshift
  
• Amazon RDS
  
• AWS IAM
  

Preferred Skills

· Hands-on experience or working knowledge of Microsoft Azure data services, including:

o Azure Data Factory

o Azure Synapse Analytics

o Microsoft Fabric

o Azure Data Lake Storage

o Azure Functions

· Exposure to business intelligence and reporting tools, particularly Microsoft Power BI or Tableau

Benefits