Exploring how to ask the right questions and measure success with bar doubts
Patti Shank, Ph.D., is internationally listed in the Top 10 Most Influential People in eLearning. He has authored several books and his articles are available on the Internet. With a strong professional background and long experience as a trainer, Patti Shank applies learning and related sciences to improve outcomes from instruction and performance interventions. Today, she discusses effective evaluation, learning objectives, and bringing skills development to the digital age.
In your opinion, what are the most common mistakes made by instructional designers when designing multiple choice questions for eLearning evaluation?
Research shows common mistakes when writing multiple choice questions, but I think the biggest mistake is Not evaluating the right thing. Most question-writers write questions by going through the content and finding things that are easy to evaluate. What we usually do Need However, evaluation means that people can make the right decisions and adequately solve the problems involved in the important tasks described by the teaching objectives.
If the teaching objectives do not adequately describe the important tasks and outcomes, the error is exacerbated. In this case, our assessments are likely to be invalid, which sets up the organization and participants for failure.
When we write multiple-choice questions to measure relevant decision-making and problem-solving skills, we are measuring the high-level skills needed to perform the tasks described for the purpose of teaching. This Should we evaluate?
Other common mistakes when writing multiple-choice questions include making it easy to guess the correct answer (no! Don’t do that!) And making it difficult to understand the question and answer choices (no! Don’t do it!). A 2013 study of thousands of Nedeau-Cayo nursing certification questions, for example, had one or more significant errors in 84% (!) Questions! [1]
Poorly written questions usually do not evaluate the intended knowledge and skills, leading to low or valid evaluation and frustration. If you use multi-choice assessment results to make decisions – such as who can move forward – poorly written questions can put your organization at legal risk.
One of the deepest areas of your expertise is writing objectives to learn to describe the desired results. What is the most challenging aspect of creating measurable objectives and why?
Too many teaching therapists don’t think it’s important for participants to know what they need to be able to do. I have been told that learning aims are old-fashioned and not a valuable use of time. But here’s the thing. If you don’t know what your participants need to be able to do, how do you know what to teach? Or if the instruction “works” to help them do it?
When learning objectives are well written, they provide clear and important guidance for designing and evaluating instruction. Without them, we’re building “content” and not “instruction.” If you are convinced that helping people build the skills they need, then content alone is usually inadequate. (And if you don’t know what people should do, the content itself may be inaccurate or irrelevant.)
Many teaching professionals do not have enough knowledge about what they are training people to do or need to know and be able to build training. They don’t necessarily understand that they are building training. How do we train when participants don’t know what they need? To do?
Learning objectives are clear statements of intent: People need to be able to perform a particular task with certain results. We Need This information!
Think of it this way: learning objectives in writing correctly is like GPS. When we input a location or address into GPS, we tell the system where we need to go so that the system can get us there. Learning objectives are similar to inputting a location or address. It provides clear information on how to get there (what and how to teach) and whether we actually got it to the right destination (assessment).
Can you tell our community a little more about your curriculum “Write good multi-choice questions to evaluate education” and who it is motivated for?
Write good multi-choice questions to assess learning A short but effective online course aimed at the skills needed to do a good job of evaluating whether the instruction achieved the desired results.
Many people think they write acceptable multiple-choice questions. Many years ago I did my best to point out my mistakes. I was humiliated and determined to improve my skills, and spent years researching how to do it. I’m glad to be able to help others demonstrate this important skill!
Research shows that many choice question writers do not know what to evaluate for multiple choice questions. As a result, many multi-choice questions measure the wrong things.
Well-written, valid multiple-choice questions that measure achievement of teaching objectives are important skills for designers. The teaching objectives for this course are as follows:
- Write well-written learning objectives to inform your multiple-choice questions.
- Analyze learning objectives and multiple choice questions to identify errors that need to be fixed.
- Write relevant multi-choice items that measure achievement of your learning objectives.
If you want more information, go to this page. If you have a team for training, you can go for a group discount or set up this course specifically for your team. We offer discounts to 3+ team members enrolling in the course at the same time or 10+ team members enrolling in the private team course.
You can join my email list by filling out a form on my site to receive a discount notice (I never, never share).
You have written many articles for the eLearning industry over the years. Are there any special articles you would like to highlight for our readers?
I Love Writing for the eLearning Industry! The articles I write teach me a lot about what kind of evidence people in our area need, the impact of that evidence. I like to share my analysis with my colleagues.
I think my favorite article is a five-part article series on when and how to use asynchronous and synchronous teaching tools. Articles are called (The Right) Teaching Modules to Provide Digital Learning: (Part 1-5). You can find them here.
I am now working on a multi-part article series on digital video and enjoying it A lot. I should have the first article soon!
What excites you most about the future of learning and development?
I’m excited to see how AI and other technologies help us learn and develop important skills. Due to constant workplace changes, maintenance and growing skills are becoming increasingly important. In many ways, though, we are still teaching people and working using 1940s methods.
I have read about how the right intangible assets are needed to make companies viable and competitive. These new insights come from innovation, which usually comes from human work. Maybe technology can help us become better inventors! For example, increasingly, complex cancer diagnoses and treatments have been added to human insights from AI. I wonder how these kinds of inventions will help us to learn and invent in the workplace.
Wrapping
We highly appreciate Patti’s skepticism in participating in our thought leader Q&A and sharing his expertise with us. If you would like to know more about Patti, please visit her author profile page, where you can find articles written for the eLearning industry, as well as her social media links.
References:
[1] Nedeau-Cayo, R., Laughlin, D., Rus, L., & Hall, J. (2013). Evaluation of object-writing errors in multiple choice questions. Journal for Nurses in Professional Development, 29, 52-57.
Homebase
Make things easier. Running a small business has never been harder. Homebase helps with free tools to track time and manage your team.