Data Engineer

Responsibilities

• Create and manage a single master record for each business entity,ensuring data consistency, accuracy, and reliability.
• Implement data governance processes, including data quality management,data profiling, data remediation, and automated data lineage.
• Create and maintain multiple robust and high-performance data processingpipelines within Cloud, Private Data Centre, and Hybrid data ecosystems.
• Assemble large, complex data sets from a wide variety of data sources.
• Collaborate with Data Scientists, Machine Learning Engineers, Business Analysts, and Business users to derive actionable insights on data quality.
• Design and implement internal processes to automate manual workflows,optimize data delivery, and re-design infrastructure for greater scalability.
• Support and work with cross-functional teams in a dynamic environment.

Requirements

• At least 5 years of experience ina data engineer role.
• Be familiar with data lake, data warehouse and data lake house architectures.
• Be familiar with MDM processes such as golden record creation, survivorship, reconciliation, enrichment, and quality.
• Experience building and operating large-scale data lakes and data ware houses.
• Experience with Hadoop ecosystem and big data tools, including Spark and Kafka.
• Experience with Master Data Management (MDM) tools and platforms such as Informatica MDM, Talend Data Catalog, Semarchy xDM, IBM PIM & IKC, or Profisee.
• Experience in data governance, including data quality management, data profiling, data remediation, and automated data lineage.
• Experience with stream-processing systems including Spark-Streaming.
• Experience working with Cloud services using one or more Cloud providerssuch as Azure, GCP, or AWS.
• Experience with Delta Lake and Databricks.
• Advanced working experience with relational SQL and NoSQL databases, including Hive, HBase, and Postgres.
• Deep understanding of SQL and the ability to optimize data queries.
• Experience with object-oriented/object function scripting languages:Python, Java, Scala, etc.
• A successful history of manipulating, processing, and extracting value from large, disconnected datasets.
• Experience applying modern development principles (Scrum, TDD,continuous integration, and code reviews).
• Proven ability to support and work with cross-functional teams in adynamic environment.
• Experience with ETL tools such as Talend Big Data, Azure Data Factory,etc.
• Experience working with Hortonworks Data Platform or Cloudera Data Platform.
• Experience with Metadata Management tools.
• Experience working on projects for Master Data Management.
• Exposure to Data Governance processes and tools.
• Proven ability in supporting and working with cross-functional teams in a dynamic environment.