Big Data System Architect
1，Key Responsibilities
Platform Architecture & Data Management: Lead the architecture design, data modeling, data development, and BI dashboard visualization for the big data platform. Establish and maintain a comprehensive data management system.
Operations & Tuning: Oversee big data platform operations and conduct continuous performance tuning.
Core Development & Optimization: Write core system code and technical documentation. Continuously optimize the system to solve real-world business problems.
Component Architecture Design: Design the application architecture for key big data components, including Spark, Hive, Spark Streaming, Flink, Kafka, HBase, Kudu, and Impala.
Computing Platform Optimization: Optimize the architecture, improve performance, extend functionalities, and perform fault analysis for both offline and real-time computing platforms.
Team Mentorship: Guide and train development team members, and share big data best practices and experiences.
2，Qualifications
Architecture & Governance Expertise: Proficient in the architecture design, construction, optimization, and operations of big data platforms. Master methodologies related to data management, governance, and analytics. Deep understanding of data quality, data standards, and master data management (MDM) methods and tools.
Hadoop Ecosystem: Familiar with the Hadoop architecture and ecosystem (e.g., Kylin, Kafka, Flink, Canal, HDFS, Hive, HBase, YARN, ElasticSearch) with hands-on project development experience.
Computing Models: Deep understanding of various offline and real-time architecture models, including their respective pros and cons.
End-to-End Data Pipeline: Familiar with the complete big data processing lifecycle, including data ingestion, cleansing, preprocessing, storage, data mining, machine learning, and data visualization. (Note: The duplicate point from the original text has been merged here).
Programming & Troubleshooting: Proficient in at least one programming language (Java or C#) with excellent troubleshooting skills.