The Importance of Direct Assessment
When conducting SLO assessment at the course level faculty should use direct assessment measures, as opposed to indirect ones. Direct measures (e.g., course papers, exams, quizzes) can demonstrate that students have or have not learned specific knowledge, skills, and/or attitudes in a course. Indirect measures (e.g., surveys, faculty evaluations), while useful in some instances, should not be used in course level SLO assessment because they involve measuring students perceptions of learning and thus can only imply that students have learned what faculty expect them to learn.
Recommended Types of Direct Assessment Tools
Faculty have a variety of tools to choose from when conducting assessment in their courses. The most common types of assessment tools for course SLOs include: embedded exam/quiz questions or final exam/quiz questions, rubrics, and pre-test/post-test evaluations. These tools are recommended for use because the data they yield allows for reasonable dialogue to occur between faculty members to improve student learning within their courses.
Descriptions of each assessment tool, their pros and cons, and information on how to create each tool can be found in the documents below.
Embedded Exam/Quiz AssessmentCourse SLO Methodology Checklist
Validity and Reliability of Direct Assessment
When creating assessment tools, faculty should keep in mind that in order to obtain an accurate measure of student learning, the tools they create should be both reliable and valid. Reliability means that the assessment tool is consistent. That is, if the assessment tool is reliable and it is applied repeatedly to the same student, that student would answer in the same way each time. There are ways to test for reliability following data collection for an assessment and the Educational Assessment Research Analyst can help you with this.
An assessment tool must also be valid, or accurate in what it is measuring, in order to be effective. Faculty should take care to ensure that their assessment tool is measuring the full scope of learning that a specific SLO covers and not measuring only a part of the SLO or an unrelated aspect. For example, if an SLO says that a student should be able to identify the 5 major conflict styles in interpersonal relationships and the assessment consists of embedded exam question, there needs to be at least one question addressing each conflict style. You can't ask students questions on two and assume that they know them all.
As you create your methodology, use the following checklist to ensure that your method is as valid and reliable as possible:
Course SLO Methodology Checklist
Embedding Assessment Questions
Whenever possible, faculty should weave the assessment tool into the course, rather than have obvious add-on tests or assignments that do not blend naturally into the course. This will make use of the actual work students produce in their courses. Faculty who are teaching the course can not only use the tool for assessment purposes, but can also give grades to students on the assignments or exams as part of the course.
Consistency in Administering Assessments
When conducting an assessment, the same assessment instrument (e.g., exam questions, assignment) should be used across a course's sections to assess an SLO. For example, if you use exam questions to assess an SLO the same SLO questions should be placed on the exams across all sections of the course to measure the SLO. Or, if you utilize a rubric to assess an essay or skill, everyone needs to use the same rubric. The instructions must also be the same to assess an SLO across sections. Why is this the case? If an aspect of the assessment differs between course sections, you can't account for as much variance. In other words, you can't tell why students did differently on the assessment. Was it because of their ability? Was it because the instructions were not clear? Was it because the assignment topics differed? There is no way to compare the assignments to tell.
Also in regard to consistency, the faculty who teach the course should decide on a time frame during which all faculty teaching the course will conduct the assessment. Why is this the case? If faculty administer the assessment weeks apart, some students may have had the opportunity to learn more than others in the course and thus, some may do better solely because they learned the information while others did not.
Faculty do not have to assess every student in every section of every one of their program's courses. Instead, they can take a sample of students. However, it is important to note that if you choose to use sampling, there will always be a difference between the scores of your sample of students and the scores of the population of students for a course or program. If your sample is not big enough, it is very likely that it will not representative of the population. When in doubt the bigger the sample, the better.
Unsure of what constitutes a representative sample when you look at size? Take a look at this document which details what the sample size would need to be in order to conduct a short exam of true/false questions for a population of a course between 50 to 1000 students.These sample sizes will provide you with data that has a confidence level of 95%. In other words, there is a 5% chance that your results happened by chance.This should give you an idea of how large your samples need to be for your own classes and programs.
Sample Size Examples
When sampling, you also need to consider the sample design. If sampling is done in a non-random method it is likely that your sample will be biased. Sample selection can easily be influenced consciously or unconsciously by human choice. If all students do not have an equal chance to be selected into the sample, it will not be representative of the students enrolled in the course or program.
How can you select a random sample? If you do not have many course sections, the best thing to do would be to add a list of all the students enrolled in your sections to excel (this can be provided to you by the Office of Institutional Effectiveness) and use the random number generator to select a sample. If you have a large number of course section offerings, you should first do a random selection in excel for the course sections and then do a random selection of students within those sections in excel.
If a random sample isn't realistic for your department, you can also use a representative sample.The goal for a representative sample is to make the sample as representative of the enrollment for a course or program as possible. When utilizing a representative sample, faculty should take into consideration: sections offered at different times throughout the day, online v face to face, the number of days course sections meet, and sections taught by full-time and part-time faculty.