21 - 27 February 2013
Twitter hashtag: #oldsmooc_w7
Professor Thomas Reeves from the University of Georgia, US and Dr. Yishay Mor from The Open University, UK.
Evaluation of learning designs is critically important. Around the world, each month sees the introduction of numerous commercially produced or locally developed learning designs, OERs, and other learning events promoted as engaging, effective, and efficient. Yet systematic evaluation of these is often lacking and evidence for their efficacy is weak. To conduct a useful evaluation of learning designs requires a "triangulation" approach whereby multiple strategies and tools are applied. This week is specifically designed to establish evaluation as a key strategy throughout the learning design process. Our specific focus will be on "formative evaluation" for the purpose of improving a learning design as it is being developed.
research and evaluation
assessment and evaluation
formative and summative evaluation
Expected time required to engage minimally with all activities is 10+ hours.
Note that if you do not have a project to evaluate, you can either evaluate any previous learning design activity you have been involved in, or you can evaluate this MOOC.
1. Begin by viewing the video introduction to the week’s activities (5 mins) (download the transcript of the introduction video)
2. Watch the Terminology Review Slidecast (10 mins) that will help you make some important distinctions in a few concepts related to evaluation (direct link: http://www.slideshare.net/tomdawguga/week-7-termreview-16347708, script of the slidecast)
4. Post a cloud to the "Evaluation experiences" cloudscape, sharing your experience of evaluating a learning design / educational project.
5. Identify a list of evaluation decisions that you anticipate having to make concerning your prototype learning design (30 mins)
Please note that anticipating decisions is not an easy task, but if you can do so in advance, your evaluation activities will be much more focused. Examples of evaluation decisions that might be anticipated:
6. Prepare a list of evaluation questions that must be answered to make the decisions identified in Task 5 above (30 mins)
Please note that each decision identified in Task 5 above may have more than one question associated with it. For example, if the anticipated decision is: “Should the content identified for this learning design be expanded, reduced, or remain the same?,” then the following questions might be addressed by your formative evaluation:
7. Post a cloud to the "Evaluation targets" cloudscape and to your project portfolio, describing the evaluation decisions and evaluation questions you have listed, and any reflections you have on the process. Comment on one or two other clouds.
8. We have started a toolbox of evaluation instruments in Cloudworks. There are further tools described at
http://www.evaluateitnow.com/downloads.html. Review and comment on these tools, and add any other tools that you are aware of to the toolbox. You can create a cloudscape for your own evaluation toolkit, and add the clouds of the tools relevant to your project to that cloudscape.
9. Prepare a draft formative evaluation plan for your learning design (90 mins)
You can find a template for an evaluation plan here: http://treeves.coe.uga.edu/edit8350/EPT.html
Please note that this plan template is probably more extensive than you’ll have time to write, but it can at least suggest some organization for your plan. Please post your plan in the "Evaluation plan" Cloudscape.
10. Comment on one or two other evaluation plan clouds (90 mins)
Monday and Tuesday
11. Review the heuristic evaluation protocol at:http://treeves.coe.uga.edu/edit8350/HEIPEP.html (30 mins)
Please note that the heuristics presented in this protocol should be modified and/or reduced to best fit the nature of your learning design. For example, if your learning design is focused on the development of an Open Educational Resource (OER) that does not include any learning assessment functionality, then the assessment heuristic (14. Assessment: The e-learning program provides assessment opportunities that are aligned with the program objectives and content.) can be deleted from the protocol.
12a. Implement a heuristic evaluation of your learning design (120 mins)
Please note that this can be done online as well as in person.
- OR -
12b. Implement any other evaluation strategy to enhance your learning design prototype (120 mins)
Please note that here are many options here, but two that may be especially appropriate would be Formative Review done by someone representative of your target learners or an Expert Review by another learning designer who is not a member of your design team per se.
You can download a Formative Review Log here: http://treeves.coe.uga.edu/edit8350/FRL.html
You can download an Expert Review Checklist here: http://treeves.coe.uga.edu/edit8350/ERC.html http://treeves.coe.uga.edu/edit8350/ERC.html
Please post your findings to the "Evaluation Results" cloudscape.
13. CONVERGE: Tuesday 26th February 12 NOON GMT Google hangout discussion. This will be a synchronous (recorded) session, which will provide an opportunity to ask questions of this week's team and reflect with others on the activities.
14. Reflect on your perspective regarding the merits and challenges involved in formative evaluation, by writing a short reflective blog post on the experience or adding your reflective account of week 7 to the Week 7 Cloudscape.
Please note that formative evaluation does not have to end here just because Week 7 is over. You are encouraged to continue to conduct formative evaluation activities so that your Learning Design can be improved as much as it is feasible. Formative evaluation is like exercise….you can always do more and get fitter.