Senior Data Engineer (Spark, Databricks, Python, MDM)
Summary
CMC Global is hiring a Senior Data Engineer to build and optimize data ingestion/processing pipelines and microservice APIs for a client's Asset Data Warehouse. Day-to-day involves master data management, data governance, and large-scale pipeline work with Spark, Kafka, Databricks, Delta Lake, and cloud platforms.
Salary: $6,500 – $8,000 per month
About the role
Our Client is looking for a Senior Data Engineer to improve data ingestion and data processing jobs and create new microservice APIs as part of ADWH (Asset Data Warehouse) enhancements.
Key responsibilities
Create and manage a single master record for each business entity, ensuring data consistency, accuracy, and reliability
Implement data governance processes, including data quality management, data profiling, data remediation, and automated data lineage
Create and maintain multiple robust and high-performance data processing pipelines within Cloud, Private Data Centre, and Hybrid data ecosystems
Assemble large, complex data sets from a wide variety of data sources
Collaborate with Data Scientists, Machine Learning Engineers, Business Analysts, and Business users to derive actionable insights on data quality
Design and implement internal processes to automate manual workflows, optimize data delivery, and re-design infrastructure for greater scalability
Support and work with cross-functional teams in a dynamic environment
About you
At least 5 years of experience in a data engineer role
Familiar with data lake, data warehouse and data lake house architectures
Familiar with MDM processes such as golden record creation, survivorship, reconciliation, enrichment, and quality
Experience building and operating large-scale data lakes and data warehouses
Experience with Hadoop ecosystem and big data tools, including Spark and Kafka
Experience with Master Data Management (MDM) tools and platforms such as Informatica MDM, Talend Data Catalog, Semarchy xDM, IBM PIM & IKC, or Profisee
Experience in data governance, including data quality management, data profiling, data remediation, and automated data lineage
Experience with stream-processing systems including Spark-Streaming
Experience working with Cloud services using one or more Cloud providers such as Azure, GCP, or AWS
Experience with Delta Lake and Databricks