Data Engineer
Summary
TCS is hiring a Data Engineer in Brazil (hybrid) to build Python-based ingestion and ETL services that pipe industrial sensor data from AVEVA PI Historian/DeltaV DCS into AWS, using Kinesis, Glue, Lambda, IoT Core, S3, Docker, and CloudWatch for real-time streaming, data-quality, and data-lake work.
TCS is seeking a Data Engineer with expertise in Python, AWS cloud services, and real-time data pipelines to design and support industrial data ingestion and ETL solutions. The role focuses on integrating AVEVA PI Historian/DeltaV DCS with AWS, building streaming data pipelines, implementing data quality controls, and supporting sensor-driven operational systems
Main responsibilities
- Develop and maintain Python-based data ingestion services to collect sensor and operational data from AVEVA PI Historian and other industrial systems.
- Design and implement real-time streaming data pipelines using AWS services such as Kinesis, Glue Streaming ETL, Lambda, IoT Core, and S3.
- Build and enhance ETL processes to validate, transform, and load high-volume sensor data into cloud storage and databases.
- Implement data quality and anomaly detection frameworks, including validation rules and BadInput detection for sensor readings.
- Design and manage data lake architectures, ensuring scalable storage and efficient data processing.
- Develop and support inspection data and large-file ingestion workflows, including industrial scan and asset data processing.
- Monitor data pipeline health, performance, and reliability using CloudWatch and automated alerting mechanisms.
- Maintain and govern sensor tag registries, metadata, and configuration management.
- Support OT/IT integrations involving AVEVA PI, DeltaV DCS, OPC-UA, and other industrial data platforms.
- Optimize SQL queries and database performance for time-series and analytical workloads.
- Deploy and manage data ingestion applications using Docker containers and cloud-native deployment practices.
- Collaborate with engineering, operations, and business teams to ensure reliable data availability for analytics, monitoring, and asset management initiatives.
Required Qualifications and Experience
- Bachelor's degree in Computer Science, Data Engineering, Information Technology, Software Engineering, or a related field.
- Minimum 5 years of hands-on experience in Data Engineering and data pipeline development.
- Strong proficiency in Python for building and maintaining data ingestion, ETL, and data processing applications.
- Hands-on experience with AWS services, including Kinesis Data Streams, AWS Glue, AWS Lambda, Amazon S3, and AWS IoT Core.
- Experience designing and supporting real-time streaming data pipelines using Apache Kafka, AWS Kinesis, or similar technologies.
- Strong knowledge of SQL and database performance optimization, preferably with Aurora MySQL or similar relational databases.
- Experience working with Docker and containerized application deployments.
- Knowledge of data lake architectures, ETL frameworks, and data quality management practices.
- Experience integrating with AVEVA PI Historian, PI Web API, OSIsoft PI SDK, or similar industrial historian platforms.
- Understanding of sensor data processing, time-series data management, and anomaly detection techniques.
- Experience with monitoring and observability tools such as Amazon CloudWatch.
- Strong analytical, troubleshooting, and problem-solving skills.
- Excellent communication skills and ability to collaborate with cross-functional teams.
Location: Brazil
Work model: Hybrid
Well-being and benefits
Health, dental, and life insurance, meal and transportation vouchers, daycare assistance, gym passes, and 24/7 support through TCS Cares (psychological, legal, social, and financial assistance). Access to cultural, wellness, and recreational programs through the TCS-SESC partnership.
Learning and Career Development
Global career opportunities through access to the latest technologies, world-class learning platforms, professional certifications, and mentoring programs designed to accelerate your growth and future-ready skills
Tata Consultancy Services is an employer that offers equal opportunities. Our commitment to diversity and inclusion drives us to offer equal opportunities to all candidates who meet our knowledge and skill needs — regardless of socioeconomic background, race, color, national origin, religion, sex, gender identity, age, marital status, disability, sexual orientation, or any other characteristic.
ID: 11067979
Skills
Sign in to read and join the discussion.