Ensuring Valid and Reliable Assessments of Student Learning Workshop

Megan Oakleaf, Syracuse University
Thursday, May 5, 2011, 1:00pm - 4:30pm
2011 LOEX Preconference

The first part of the workshop focused on outcomes and performance assessment. The second part covered rubrics.

Outcomes
Megan Oakleaf briefly discussed the necessity for "clear, meaningful, transferable, learning outcomes." How will students be able to use what they learn in other contexts. Ensure that the outcomes are relatable, but not overwhelming.

There are a lot of different formulas for writing outcomes, but all good outcomes begin with active verbs. See Assessment-as-Learning.

Performance Assessment
Oakleaf stressed active learning. She briefly discussed the "Understanding by Design" approach before discussing performance assessment. Performance assessments “focus on student’s tasks or products/artifacts of those tasks.” These assessments “simulate real life application of skills.” I was surprised to learn that your instruction/learning tool can also be your assessment tool. It seems so obvious, but I had never thought of it that way. For example, observing students completing a class activity such as a database search can be an assessment. Develop a checklist and check off as a student meets the criteria. She used the example of tallying how many students locate an appropriate article. In our introductory workshop, we could keep track of how many students are able to locate on book on their own in our find-a-book activity.

Rubrics
Rubrics “describe student learning in 2 dimensions:” indicators/criteria and “levels of performance.” Oakleaf introduced us to the RAILS (Rubric Assessment of Information Literacy Skills) project: www.railsontrack.com.

Rubric Creation Process

1. Reflecting: "why did we create this assignment/assessment?" "What happened the last time we gave it?" "What is the relationship between this assignment/assessment and the rest of what students will learn?"
2. Listing: "What specific learning outcomes do we want to see in the completed assignment/assessment?" "What evidence can students provide in this assignment/assessment that would demonstrate their learning.". "What are our expectations of student work? What does it look like?"
3. Grouping and labeling: "Can we group our brainstorms in categories?" "How can we label them?" The labeled groups are now the "criteria."
4. Creating: Draft the performance descriptions. Define the highest level of or best possible student performance. Define the worst and other developmental levels as needed.

In order to avoid the most common rubric design flaws, describe what a student at that level ‘looks’ like.

Rubric Norming Process
Oakleaf recommends norming rubrics when multiple graders will be using the same rubric. This will limit discrepancies among evaluators.

1. Think aloud about an example. Criterion by criterion, share how you came to your performance rating.
2. Raters independently evaluate examples.
3. Raters come together to identify similarities and differences in scoring patterns.
4. “Discuss and reconcile inconsistent scores.”
5. Repeat steps 2-4 new samples until consensus.

Closing the Loop
Based on your assessment, “Enact decisions to increase learning.” Assessment is useless if you don’t do this.

Student Response Systems in Library Instruction

A few months ago, Evelyne Corcos and Vivienne Monty published a study of student response systems in library instruction. Their article, “Interactivity in Library Presentations Using a Personal Response System” appeared in EDUCAUSE Quarterly. Personal response systems (PRS) (aka student response systems or audience response systems) allow students to use wire-less clickers in order to respond to instructors’ questions. An instructor poses a multiple-choice question and students respond by clicking the appropriate button on wire-less, handheld “clickers.” The instructor can display the live results to the class or to him/herself.

In Corcos and Monty’s study of personal response systems (PRS), York University students participated either in traditional library instruction or PRS-enabled instruction. The librarians selected PRS as a means of increasing student interactivity in library instruction. Students in the PRS classes ranked instruction as more enjoyable and engaging and better presented and organized. The librarian in the PRS classes enjoyed knowing students were active by their clicker responses. Nevertheless, PRS did present challenges in time involvement and technical difficulties. Corcos and Monty recommend “a standardized routine for setting up the PRS equipment and software.”

Pros

• Immediate feedback
  
• Opportunity for formative assessment
  
• Tailor instruction based on responses
  
• Identify areas of low comprehension
  
• Anonymity reduces student embarrassment
  

Cons

• Technical difficulties
  
• Pre-set questions reduce flexibility
  
• Requires more preparation
  

The authors don't list this as a con, but it seems to me that keeping up with the clickers could be a hassle. How do prevent students from accidently taking the clickers home? Anyways, I would still love to try them.

Assessment of Information Literacy Programs

From Immersion '06, July 31, 2006

Faculty Lead: Debra Gilchrist

For assessing a program, as opposed to student learning, Deb reframed the 5 questions from her “Assessment-as-Learning” sessions.

1. “What do you want the instruction program to be able to do? What does it do when it meets student needs and/or supports student learning?” (Outcome)
2. “What factors does the program demonstrate or exhibit if they are meeting the outcomes? What will be happening?" (Indicators)
3. “How will you gather your data or evidence?” (Assessment)
4. “How will you determine if you have achieved your outcomes?” (Criteria)
5. “How will the discussion or evaluation of the data occur? Who will be involved?” (Change)

It is important that the assessment leads into a cycle of continual improvement. Deb’s cycle of Outcomes/Indicators/Assessment/Criteria/Change reminded me of the Plan/Do/Check/Act cycle used in my organization. Deb recommends using ACRL’s “Characteristics of Programs of Information Literacy that Illustrate Best Practices” when establishing program outcomes.

Assessment-as-Learning

During Immersion, Debra Gilchrist presented 3 workshops on assessment, 2 general sessions on assessment-as-learning and 1 on program assessment. My notes from the first 2 sessions are below. I will post my notes from the session on program assessment later.

Assessment as Learning Pt. 1 July 29, 2006
Faculty Lead: Debra Gilchrist
Deb’s 2 rules for assessment: 1) Work backwards from what you students to be able to do and 2) Assessment is about student learning.
Deb’s “5 questions for assessment design:

1. “What do you want students to be able to do?” (Outcome)
2. “What does the student need to know in order to do this well?” (Curriculum)
3. “What activity will facilitate learning?” (Pedagogy)
4. “How will the student demonstrate the learning?” (Assessment)
5. “How will I know the student has done this well?” (Criteria)

Deb’s outcome writing formula: Verb phrase + “in order to” + impact phrase
An example from Deb: "Develop topic relevant vocabulary in order to search databases with maximum flexibility and effectiveness."
Bloom’s Taxonomy is helpful in writing the verb phrase that describes what you want the student to be able to do. The impact phrase explains why you want the student to be able to do it.

Assessment-as-Learning Pt. 2 July 30, 2006
Faculty Lead: Debra Gilchrist
This session began with a brief review of the outcome formula. In the formula, the verb phrase is the skill and the “in order to” statement is the application or impact. Deb believes outcomes are fully measurable. Recall from the “good learning outcomes” slide that they are measurable/“judgeable.” Measurable outcomes are quantitative and judgment outcomes are qualitative. A judgeable outcome is where the instructor can see that the student has improved (I am thinking this means behavior wise). Assessment test like SAILS and ETS can provide a benchmark. The best assessments include a critical thinking element. Debra than shared some documents demonstrating IL assignments and outcomes that librarians have helped faculty develop and self-assessment students utilize in library workshops.

At Pierce College, the library provided a faculty workshop where they analyzed assignments with faculty. The faculty submitted assignments in advance and the librarians identified what IL concepts students needed to know in order to complete the assignment. They then worked with faculty to revise outcomes.

At Pierce College they assess only one out of five outcomes each year so that each out come will be assessed at least once over an eight year period. More important outcomes may be assessed more often than less important ones. At first they only assessed 1 in 10 sessions (10%), now they assess 1 in 4 (25%).

PS. Be sure to check out Pierce College's Library Instruction Program. Notice that Information Competency is on of the college's 5 core abilities. Pretty cool, huh?