Assessment Analyst

Duties and Responsibilities

Assessments

· Work with assessment team to design assessment blueprints

· Work with assessment developers to align content within assessment blueprints

· Write assessment design training documents for various groups using appropriate style

· Document technical aspects of exams including item performance, practice analysis procedures, and standard setting reports

· Collaborate with staff to improve and automate processes for assessment activities

· Create research programs to evaluate assessment characteristics in the field

· Manage and operate the system for processing scores in a timely fashion

· Provide backup support for psychometricians in tasks as needed

Data Management and Analysis

· Collect, clean, validate, and transform large datasets from certification systems, assessment platforms, CRM databases, and external sources

· Conduct descriptive and inferential statistical analyses to support certification program validity, exam performance reviews, and candidate experience improvements

· Build dashboards, reports, and data visualizations for internal and external stakeholders

· Exhibit Data Stewardship: Understand where data is, how to access it, and how it can be used

Reporting and Communication

· Develop written and visual reports for leadership, committees, accreditation bodies, and partner organizations

· Translate complex analytical findings into clear, actionable recommendations

· Present data insights to cross-functional teams including, but not limited to, Education Requirements, Marketing and Strategic Communications, Finance, and Information Technology (IT)

· Assist with the design and development of Annual Reports for assessment programs

· Serve as a data liaison between departments

Data Infrastructure and Quality

· Partner with IT to improve data pipelines, data quality, and overall system reliability.

· Document data definitions, processes, and best practices

· Ensure compliance with data privacy standards and accreditation requirements (e.g., NCCA, ISO/IEC 17024)

Assessment Content and Eligibility Materials

· Contribute to periodic practice analysis projects for assigned programs, including survey development, data analysis, and report writing

· Contribute to the development and maintenance of the item bank quality metrics

· Assist with the development of procedures for a more standard presentation of practice analysis survey data where possible and incorporate documentation on those processes into the Practice Analysis Manual

Assessment Scoring, Analysis, and Reporting

· Process exam results and conduct quality control (QC) for a variety of programs

· Conduct exam results processing, scoring, and QC for all programs

· Maintain procedures manual and other documentation required for exam processing and scoring

· Analyze exam results and collaborate with department staff in production and dissemination of annual examination summary reports and technical reports

· Recommend enhancements to scoring procedures and reports

· Analyze data to support department(s)

· Collaborate to create software tools that directly facilitate or automate psychometric work responsibilities

Miscellaneous

· Continuously watch for, communicate, and seek validation of areas of quality improvement by initiating discussions and participating in evaluation of improved processes

· Write, review, or edit other ARRT documents as requested, including website material

· Assist with or prepare memos and reports for presentation to the Board of Trustees

· Assist with standard setting studies, as needed

· Support organizational priorities through active participation in ARRT Projects as assigned.

· Proactively manage and deliver assigned ARRT project work.

· Other duties as assigned and modified as business needs dictate.

Minimum Qualifications:

· Master’s degree or higher in psychology (I/O, cognitive, education, research methods, assessment design), data science, statistics, economics, or related field

· Experience with data analysis and accessing data from a database (SQL)

· R programming language

· Proficiency in Excel

· Two years of experience, preferably within a non-profit, certification environment; or relevant field

· Strong understanding of statistical methods and data integrity practices

· Familiarity with survey design and analysis

· Experience and/or education focused on working with assessment data (item-level statistics, exam forms, psychometric outputs, etc.)

Preferred Qualifications:

· ICE-CCP certification

· Experience with data visualization tools (Power BI, Tableau, or similar)

· Understanding of accreditation frameworks (NCCA, ANSI/ISO 17024)

· Knowledge of CRM systems

· Assessment development experience including requirements and content specification

Skills and Abilities:

· Excellent interpersonal, verbal, and written communication skills with demonstrated ability to communicate in a clear, concise, and professional manner with the ability to collaborate across groups to build and maintain relationships

· Ability to evaluate and establish priorities

· Research, interpret, explain, present, and apply complex procedures, rules, and regulations

· Communicate and manage detailed, sensitive, and confidential information appropriately

· Proficiency in psychometric concepts, tools, techniques, and best practices

· Adaptability and flexibility, with the ability to respond to changing needs of the position, department, and organization.

· Professional development participation to remain current with industry best practices.

· Proven ability to apply critical thinking and problem-solving skills to analyze, simplify, and translate complex business problems and apply the appropriate level of planning, organization, and structure to drive results to multiple levels in the organization

· Demonstrated ability to utilize and understand Microsoft 365 office products

· Experience with R, SQL, IRT calibrations, item analysis software applications (e.g., WINSTEPS), assessment engineering

· Familiarity with multiple data collection techniques including, but not limited to, operational, survey, etc.

· Openness to learning machine learning models

Physical Demands:

Close Visual Acuity: Preparing and analyzing data; viewing a computer screen; and extensive reading

Fingering: Typing, or otherwise working, primarily with fingers

Hearing: Ability to receive detailed information through oral communication, and to make fine discriminations in sound.

Reaching: Extending hand(s) and arm(s) in any direction.

Repetitive Motions: Substantial movements (motions) of the wrists, hands, and/or fingers

Sedentary Work: Sitting most of the time; occasional walking and standing

Talking: Spoken word activities can be relayed accurately, loudly, or quickly