Evaluating American Sign Language Performance Case Study

Susan Rose, an associate professor in the Department of Educational Psychology on the Twin Cities campus, and Simon Hooper, an associate professor in the Department of Curriculum and Instruction on the Twin Cities campus, are designing, developing, and evaluating an Internet-based tool to facilitate evaluation of American Sign Language (ASL) students' signing proficiency. Materials generated through the project might also be used to develop an ASL evaluation process training program, and the software might be used to evaluate students' performance in related and other language courses.

Instructional Goals

Because ASL has no written component, and postsecondary-level instructional materials are virtually nonexistent, instructors face many difficulties teaching ASL and evaluating students' performances.

University of Minnesota ASL instructors teach four classes of at least 24 students each semester without teaching assistants. The instructors have required for mid-term and final exams that each student locate a video camera, record him- or herself signing about an assigned topic for 15 minutes, and submit the video on tape to the instructor.

In addition, students who want to take ASL to meet the world languages studies requirement must pass a competency exam called the Sign Proficiency Interview (SPI). For each student, an interviewer must be hired, a video recording room set up, a video-taped 25-minute interview conducted, and the recording evaluated by the interviewer and one other person. Approximately 200 students take the exam each semester at a cost of approximately $7,000 to $9,000. In addition, the validity of the process has been questioned.

According to their 2002 TEL grant proposal (PDF), the investigators hope their tool will meet the following goals:

"improve cost-effectiveness;
increase efficiency;
improve assessment reliability; and
enhance student[s'] access and opinions of the learning experience."

Technology Strategies

The investigators wanted students to use computers and digital video cameras to do the following:

record their performances on photo recognition, story recall, story completion, and other tests;
store the recordings on a central server;
access the recordings with any Internet-connected computer
re-record their performances until satisfied;
submit the recordings for evaluation; and
build personal signing portfolios.

Evaluators would use corresponding equipment to complete the following tasks:

access the recordings,
generate feedback, and
report students' grades.

As they began to develop the tools, they quickly came up against a series of basic questions about quality, frame rate, file size, and the like. They found that they would need to orient the students prior to assessment in three phases:

video and photo interpretation,
camera alignment, and
recording.

They wanted the design to motivate the students, so they set up the following design parameters:

use minimal text,
keep the design simple,
enable users to to do everthing with a single click,
create fluid transitions between phases so students can focus on substantive content,
demonstrate how to complete the test to acclimate students to the process, and
constrain the testing process to a single window on the computer screen.

Learning Outcomes

The investigators plan to evaluate the project by conducting the following:

formative evaluation tests of the effectiveness of the software,
a cost-benefit analysis to determine financial implications, and
a survey and focus group to assess user satisfaction.

Related Resources

You need Acrobat Reader software to view portable document format (PDF) files. You can download it for free from the Adobe Acrobat Reader page.