Big Data Engineer
Summary
Designs and maintains ETL/ELT pipelines and real-time dashboards for industrial data, using PySpark, Hadoop/Cloudera, and BI tools to drive operational efficiency and safety in sectors like oil & gas and manufacturing.
About the job Join our Team - Your career journey starts here - not just a job, but a future. Our Potential Opportunity INTECH Automation Intelligence is seeking a Big Data Engineer to join our dynamic team. This role is pivotal in supporting Industrial Digitalization Solutions across industries such as Oil & Gas, Petrochemicals, Utilities, and Manufacturing. Design and operate end-to-end big data and analytics solutions building scalable ETL/ELT pipelines, integrating industrial and enterprise data, optimizing Hadoop/Cloudera platforms, and delivering real-time, predictive dashboards that drive operational efficiency, safety, compliance, and data-driven decision-making. INTECH's commitment to providing cutting-edge Digital Solutions ensures you will contribute to transformative projects that enhance operational efficiency and productivity. Our Digital Solutions are a testament to our dedication to driving innovation and optimizing industrial processes, reflecting our focus on excellence, reliability, and overcoming industry challenges.
What You’ll Lead and Deliver
- Design, develop, and maintain robust ETL/ELT pipelines for both pre-production and production environments.Ingest data from diverse sources, including industrial sensors (pumps, boilers, storage tanks), relational databases (SQL, Oracle), and enterprise applications.
- Implement data integration and transformation logic using Scala, PySpark, Sqoop, and Apache NiFi within the Cloudera/Hadoop ecosystem.
- Manage and schedule data workflows using Oozie or similar orchestration tools.
- Optimize data storage and query performance using Hive, Impala, and HDFS for efficient analytics.
- Develop interactive and real-time dashboards in Power BI and Tableau to monitor critical business and operational KPIs.
- Translate complex technical datasets into user-friendly visualizations for stakeholders, enabling data-driven decision-making.
- Design dashboards for specific use cases such as:
- i) Real-time monitoring of sensor data (pumps, boilers) to improve plant uptime and save costs.
- ii) Track safety permits (ESWP), monitor workplace hazards (SHER, risk scores), and perform root-cause analysis on incidents.
- Support the development of predictive models, such as Hazard Prediction dashboards, by preparing and engineering the necessary data features.
- Perform data analysis and ad-hoc reporting to answer complex business questions related to plant manufacturing, energy, and safety.
- Ensure data accuracy, consistency, and reliability across all reporting and analytics platforms.
- Assist in the maintenance, configuration, and performance tuning of the Big Data platform (Cloudera) and BI servers (Tableau/Power BI).
- Collaborate on the integration of industrial systems (e.g., OSI PI System) with the big data lake for advanced analytics.