Assessment Practices in Online Contexts (4.0)
What Does the Literature Say?
For both the learner and the instructor, success is maximized when opportunities for reflection, feedback and forward planning are done on an ongoing basis. Ongoing assessment practices are integral to planning for learner and faculty success while using online technologies.
Learning in an online environment is devoid of many of the contextual cues that we depend upon for personal feedback on "how we are doing" as teacher or learner. There are no visual cues such as body language; no auditory cues such as the tone of voice in which a response is given; no glazing over of the eyes nor instances of wild-eyed enthusiasm apparent from interacting with text-based instruction. Thus, it is incumbent upon the instructor to build in strategies to both give and receive feedback and integrate such strategies into each lesson.
As well, since some point to the issue of learner identity as a challenge to be addressed, it is a good strategy to think about how assessment practices can address this issue head-on.
To appreciate the strategies we will suggest for assessing both learners and the effectiveness of a course using online technologies for learning there are some basic considerations that need to be taken into account. The first involves a lot of public perceptions and concerns that are often expressed about learning online. The main one of these being that because you don’t see the person completing the assessment then someone else may be doing it instead (Paloff & Pratt, 2009). While it is true that one may not see the learner actually completing the work, particularly in the case where a course is fully online, there are actually very effective ways to check if the same person has been doing the various assignments and is the same person who is, for example, contributing to the online discussion. Largely this is because the online environment offers a permanent trace of activity, allowing both the learner and instructor to observe contributions from different times in the course. From this activity trace, one can readily note differences in style, vocabulary or grasp of concepts that could indicate a problem.
The second issue affecting the choice of assessment and evaluation practices is the underlying pedagogy that directs the online course, or course component. From a constructivist perspective, the emphasis is on assessments that highlight the pedagogical value of relationships, inquiry, invention, and the construction of understanding through collaborative work and discussions (Rovai, 2001). However, when instructors first teach using online tools there is a tendency for instructors to continue to use the same assessments as those from the in-class context, particularly a focus on exams and quizzes. While these can still be used as part of online technologies for learning, there are a number of other effective strategies you might wish to incorporate that may better tap into areas of learning online, and we will look at such options later in this section. That does not mean you cannot use online exams and quizzes as there are even more choices available today through the Internet. However one might consider using quizzes also for learners to use to self-test, not simply as a summative assessment tool for the instructor, and, further, to include those as one of a variety of assessment measures. Paloff and Pratt (2009) suggest a range of assessment strategies that are more suited to the online context, in that they take advantage of the features the online environment offers, such as the time to reflect, collaborate and look at growth over time.
Their principles for online assessments are:
- To provide continuity between instruction and assessment by using learner-centred assessments which complement learner-centred activities.
- Construct course with a wide range of learning activities and assessment measures that will appeal to a range of learners and at the same reduce the opportunities for plagiarism and cheating.
- Encourage and assess learner contributions to discussion.
- Use rubrics to establish performance expectations and to provide a meaningful basis for self-assessment or teacher assessment.
- Always provide prompt feedback on assessments and assignments. This both encourages engagement and reduces anxiety among learners.
- Treat online quizzes and tests as open-book learning experiences and design them with this in mind. This will reduce instructor anxiety about cheating!
- Use a variety of more extended measures such as projects, learning portfolios, self-assessments and ongoing assessment of discussion contributions.
So far we have looked at considerations applied to assessment of learners. In terms of assessment of courses, many higher education courses tend to be evaluated through surveys of student attitudes usually at the conclusion of the course (Roberts, Irani, Telg & Lundy, 2005). Further, classroom-based classes often focus on the instructor’s performance. Paloff and Pratt (2009) suggest that in a more learner-centred context course assessments should focus on what the learner found helpful in terms of opportunities for learning. In addition to an evaluation of the instructor performance they suggest a focus on some or all of the following issues in a summative course evaluation:
- Perception of the overall online course experience.
- Orientation to the course and course materials.
- The content itself—including the quality of the materials and the quantity—remember learning online takes longer.
- The value of the discussion and the interaction both with peers and the instructor.
- Self-assessment of participation and performance in the course as well as the degree of contribution to the learning of others.
- The technology or technologies used in the course and their ease of use and how they contribute to learning.
Adapted fromBrett, C. & Smyth E. (2010). Integrating Online Technology for Learning. Module 6 of a series: OISE Higher Education Teaching Series. Modules for Faculty Development. Continuing Education, OISE, University of Toronto.
In the rest of this section we will provide examples of different assessments you might use in online technologies for learning and how to integrate them into your course.
How Do I Construct Effective Online Assessments of Learners?
Constructing effective online assessments requires that you keep the constraints and features of the online environment in mind, as well as some of the features made possible by the online context, such as the opportunity to maintain a permanent record of contributions over time.
In the following activity, some assessment strategies are described and you can think of how you might use:
Selecting Assessment Strategies
| Strategy | Purpose | How do I do this? | Where might I use this in my course? |
|---|---|---|---|
| Online Quizzes |
| Explore the Learning Management System available to you for test and quiz utilities | |
| Student self assessment |
| Integrate these strategies at the end of sections or units | |
| Learning Portfolios |
| Work with students to assist them in selecting exemplars of their work throughout the course | |
| Class contributions (teamwork) |
| Plan on an ongoing basis to construct two-way feedback | |
| Responses to discussion questions | Depth, clarity demonstration of conceptual understanding | Utilize rubrics as a standard |
Constructing Effective Online Assessments
The instructor can improve the teaching and learning environment by employing some simple strategies that will make for effective learning. What is critical is that if participation is being evaluated (e.g. as part of a participation grade) that criteria are clear and that the instructor can provide the evidence of how the decision was arrived at:
Give feedback right away on amount and quality of online responses. The key qualities you are looking for can be clarified by providing a rubric for online responses and making online contributions part of the overall grade. If one thinks of writing as the online equivalent to speaking, and reading as the online equivalent to listening, this becomes a way to think about a participation mark, as a separate components beyond those assigned to individual assignments. It is important to explain this to students so the basis for evaluation is clear. Feedback should also be regular so that learners can tell where they are.
Clearly state what your expectations are for each aspect of the required work: the use of technology, the nature and scope of the assignments, and so on.
For large project type assignments include a series of timed and connected stages toward completion. This also supports student time management, instructor monitoring of student work, and reduction of plagiarism.
Feedback Strategies: Communication
Communication is the key to effective assessment. This can be done both within and outside of the particular online environment used. Direct communication with an individual in the form of an email or telephone call may be more appropriate than sharing an evaluative comment with the entire course. Use discretion here.
Ask yourself: Is This an Issue Common to More Than One Learner?
If the answer is yes, share the comments more broadly. Within the first weeks of a course, the instructor can do a lot of work behind the scenes – advising learners on their progress, encouraging them to contribute a more substantive commentary; reminding them that quantity is no substitute for quality. Many online technologies come equipped with quantitative measures that can be used to reinforce the need for ongoing participation and can assist the instructor in ensuring that learners do not get lost in cyberspace and disappear into the ether.
It is a good idea to keep two independent pieces of contact information for students, in the event that your online technologies for learning environment hiccups or collapses for a day or two. It is important that you are able to contact the learners to reassure them that there is a system problem. As in a face-to-face classroom, there should be no surprises with respect to learner progress. In planning assessment strategies, the instructor should be guided by assessment policies of the institution sponsoring the course. This is especially sensitive in the domain of ‘class participation’. In a text-based online environment, that requires carefully crafted analytical submissions, regular participation more closely resembles the submission of short, pithy essays than oral participation in a seminar. This should be reflected in the evaluation scheme presented.
Assessing Student Progress
Outline expectations about the length, the technical level (e.g. spell checked etc.) and the content (see below) of online responses. Then give feedback early on, perhaps through an introductory activity, on the quality of online responses so that people have a concrete sense of whether they are meeting expectations.
Assessments should be regular so that students know how they are progressing, and work is not left to a large assignment at the end.
For large project-type assignments include a series of timed stages toward completion.
A way to assess whether the online participant is the one actually producing the work (a common concern levied against online courses) is to make the assignments progressive. For example a large final project that involves incremental implementations of the project components for a series of assignments.
What counts? It is important to include a component for class participation to the conference.
You can either define a set of criteria ahead of time, or have a consensus-gathering activity about what makes a good note.
Examples:
- What kinds of notes do you like to read?
- What is it about a note that makes it useful and why?
- When do you find a note not useful and why?
- What criteria might we want to establish for notes?
- This note helped me learn…
- This note was socially supportive
- This note contained both helpful content/ideas and contributed to the ongoing discussion.
Such an approach allows students to self-assess the quality of their own contributions.
For the instructor to evaluate the quality of the online contributions themselves it is helpful to consider some possible criteria to use in judging their quality, for example:
- Does the entry deal with an important conceptual issue relevant to the course?
- Does the entry use concepts from the readings?
- Does the entry acknowledge relevant peer input already present in the discussion?
- Does the entry add value to the discussion through adding new knowledge, a useful example, or a pertinent summary.
- Does the entry make an advance on the original idea (such as by integrating a number of ideas and coming to a new insight)?
Such a set of criteria can be given to students at the beginning of the course to guide their writing.
Rubric for Participation
This is a sample rubric for participation that can be modified for your use.
| Score | Criteria |
|---|---|
| 0 | Logs in to the course but does not read the notes or contribute. |
| 1 | Logs in to the course. Reads some of the notes. Occasionally contributes when specifically required to do so. Often is present online as an observer rather than an active participant. Note: this student may be reading only and slower to participate. Be sure to check if when they do comment it is thoughtful and based on valid observations: you can grade them higher if they seem to learn better this way, and even reach out to them in a Private Note or message. |
| 2 | Logs in to the course. Reads many of the notes. Participates as required. Presents minimal input and that is primarily derived from assigned readings. Does not offer much beyond evidence of collaborative or developed ideas. |
| 3 | Logs in to the course. Reads most of the notes. Active and consistent participation in the discussion. Posts indicate that all readings are considered in the responses. Posts consider the contributions of others. |
| 4 | Logs into the course. Reads most of the notes. Active and consistent participation in the discussions. Posts indicate that the learner has gone beyond readings assigned and integrates other knowledge and experience in the posts. Posts demonstrate a high level of analysis of the contributions of others in a thoughtful and respectful manner. |
State Your Expectations: An Example
Clearly state what your expectations are. It is useful if you give clear guidelines for learner participation in your online environment.
Here Is an Example of How These expectations Could Be Stated.
It is my expectation that you respond AT LEAST three times in the course of the session. The first time will be your RESPONSE to the tasks – the questions that guide your response to the assigned readings. The second and third posts will be your response to each other. You should plan a response on the weekend as many people tend to post their information on Saturday and Sunday. Please post your initial response to each task as a build on to the tasks in this view, in a well-crafted response of a maximum of 250 words per task. It should be well titled — not "I AGREE!" After this initial post, you are required to post at least two additional well-crafted responses to the notes posted by your co-learners. These should be done after the majority of initial notes are posted to the course. Your response should be thoughtful and reflective.
Expectations: Example
These are my expectations for the entries you will post:
- compose your responses to the articles in your word processor
- proofread your response
- format, inserting HTML code (optional)
- save your response in a safe place – i.e. on your c drive or on the cloud
- before going on to the next step
- copy your text onto your clipboard
- with your wordpro running, open the discussion board
- go to either New Note or Build On
- using cut and paste, paste it into the text box
- insert links, graphics (optional)
- insert a title which reflects your comments
- select Contribute
- review how it will appear
- edit it using the Edit command as necessary
- select Contribute again
I will encapsulate the discussion and comment with a synopsis to your responses.
If you are "absent" I will send comments to you via personal email if I am concerned about your 'attendance' etc. So please do two things: check the email address I have for you in the email list and check that email during the course.
Scaffolding
For large project type assignments include a series of timed stages toward completion.
One effective strategy to assist learner success is to scaffold the assignments.
Make all the assignments nest one into the next.
Culminating Assignments: Example
Here is an example:
Focus all course activity toward a culminating assignment.
The stages might include:
- proposing a topic and gathering preliminary resources
- refining a topic and generating a research question
- building a focused bibliography
- reviewing an article drawn from that bibliography
- reporting progress throughout the course
- writing a final paper
Each of these stages can have a deadline date at a point in the course that spreads the workload.
If the project is a group one, you could have learners submit evaluations of the group process by email. That way you as the instructor can be apprised of any issues that may have transpired during the group experience and take that into account in evaluating the project.
An example of such an assessment follows:
Group Assessment
One of the biggest challenges in collaborative environments is evaluation. Should a collaborative effort be given a group mark or should it be given an individual mark? Individual marks reflect individual accountability and may encourage cooperation, group marks encourage interdependence but may also encourage free-riding. The assignments in this course will be assessed on both an individual and a group basis.
Group Evaluation – Progress Report
Learners often feel that their partners may unduly benefit from their work. Research indicates that if students are able to have multiple points of peer assessment – free riding will be decreased. As you reach the half-way mark on this assignment – each member of the group should send [instructor] a brief evaluation indicating how the work is progressing. Please consider these questions in this progress report. This will comprise part of your overall assessment.
- What action has each member taken that is helpful to the group? (10%)
- What action could be taken to improve the group? (10%)
Group Evaluation – Final Evaluation
Upon completing the project each member of the group will submit another brief report indicating how the group was able to function together.
- What were the groups overall accomplishments? (5%)
- What was my individual role in furthering this project? (5%)
- What could be changed in the future to make group work more
effective? (5%) - Project Evaluation (overall worth 35% of the course mark)
- The group mark will comprise 15% of the project mark, the individual mark the remaining 20%.
How Do I Assess the Effectiveness of a Course?
It is a good idea to ask for feedback during the course: reactions to readings, to assignments, and so on. It is also a good idea to share with your learners an approach to the course that is open to accepting suggestions for change and improvement. Encourage them to share their positive and negative reactions with you. To the extent that they feel listened to and able to express their views, they will usually be more willing to try things and engage themselves in the material. This is particularly important for adult learners.
You can also compare quality/depth face-to-face assignments (from earlier offerings of the course) and the online ones to assess any differences.
You can build in an assessment of assignments into the assignment evaluation itself. For example, in a self-assessment of weblog entries, one of the authors of this module asked her students to submit an email justifying their self-chosen weblog mark based on a rubric provided by her:
Beginning Rubric for Weblog Reflections
(4 is highest, 1 is lowest)
4. Entries were regular and characterized primarily as very reflective (i.e., reacting to ideas from the course materials, elaborating ideas from readings and from others’ contributions, and contemplating or extending ideas from readings and from others’ contributions (value-added contributions). As well, entries monitored the learner’s own growth in understanding, and monitored learner understanding by providing appropriate evidence and justification.
3. Entries mainly characterized as reflective (i.e., reacting to ideas from the course materials, elaborating ideas from readings and from others’ contributions,).
2. Entries mainly characterized by description and limited depth of reflection (i.e., reacting to ideas from the course materials).
1. Entries were infrequent and characterized primarily by description and limited depth of reflection (i.e., reacting to ideas from the course materials).
In that justification they discuss their reaction to the assignment and the technology and how this changed, or not, over the duration of the course. Such feedback provides the instructor with a really in-depth understanding of how the learners are understanding aspects of the course.
Course Evaluations
Course evaluations can range from the very formal such as structured questionnaires to learners to complete a “Start, Stop and Continue” entry at the end of the course.
This can be done individually by email, or anonymously online. One of the authors uses a discussion entry in which all the class are authors. She provides the following instructions: the more open-ended model.
An example of an open-ended strategy is to ask:
While I would really appreciate it if you would complete the formal course evaluation [URL] it is also helpful for me to get a sense of what people think of the various aspects of the course in a more detailed way. In this collaboratively edited note, I don’t know who has contributed so there is anonymity (unless you want to declare yourself!). Just add your comments under the appropriate heading below. These comments are taken into account as I review the course for the next offering. You can comment on any aspect of the course that you wish—the instructor, content, readings, online tools and so on.
Generally, remember to ask questions in your evaluations that look at a range of the course learning experiences, including interaction and collaboration, and beyond the content and instructor performance.