The other day we recorded a short video with John Hughes for his and Katherine Bilsborough’s YouTube Channel, Writing ELT Materials on #itemwriting for ELT tests and exams.
Much like we had established the 5 key principles typically mentioned when we talk about #assessment (i.e. reliability, validity, fairness, practicality, and backwash) with Jim Fuller as well, we decided to focus on 5 key best practices when writing ELT questions (aka ‘items’).
Here they are.
1) We must understand validity (in light of Communicative Language Teaching) and test the new language learned (i.e. its meaning, use, form and appropriacy) for communicative, real-life purposes, not just knowledge about the language (i.e. knowledge of grammatical terms and mere form, etc.).
TIPS:
• the correct answer shouldn’t be ‘inferable’ (i.e. all the options should be plausible)
• the candidate/student should be able to answer the question without having to make any (e.g. mathematical) calculations to get the answer
2) We must be clear about our construct (“a concept that is defined so that it can be scientifically investigated” (=operationalized and measured/inferred). If we look at an item and aren’t immediately sure what it is trying to test, chances are it still needs work.
TIPS:
• the question needs to be ‘keyable’ (i.e. there has to be one correct answer only)
• the questions need to follow the same order as the text/audio
3) We want to also be clear about whether we’re writing ‘systems’ (i.e. grammar/vocab) or ‘skills’ (i.e. listening, reading, writing or speaking) items.
TIPS:
in grammar and vocab items:
• all the options should be consistent with the scope of your target language
in skills items:
• you should properly ‘anchor’ the stem in the text/audio (using predication, e.g. *’Concerning the school’s courses…’ vs ‘Concerning the school’s courses Mark says…’
4) It’s also important to be mindful of our language. It should generally be
• simple
• correct (and consistent e.g. AmE vs BrE)
• necessary
and
•…on level!
TIPS:
• avoid ‘phrase-lifting’ from the text/audio (…and try to even avoid using the same root word, e.g. happy/happiness)
• write independent distractors (=options) whose meanings don’t overlap or contradict each other
5) We should also be consistent about format. Some tips that make an item ‘user-friendly’ include:
TIPS:
• your options need to form a set (i.e. be similar to one another in terms of the grammatical structures they contain and/or their word count, register, connotations, etc.)
• put any repeated words or chunks in the stem
EXERCISE:
How would you categorize the following tips? Are they mostly about
–validity
–construct
–item type (i.e. language or skill)
–language
or
–format?
[NB: The answers to this exercise aren’t meant to be ‘keyable’ because they do overlap in some cases and are subjective. ;)]
TIPS:
• if you’re writing more items in a batch, make sure you keep them varied (e.g. don’t only write declarative statements using the first person singular, but negative sentences, yes/no questions or wh- questions too, as well as use other pronouns (e.g. he/they).
• don’t start an item with a blank
• don’t write ‘Which one is NOT an …’ kind of items
· check that no language in the item is of higher level than what the item tests (use the Pearson Global Scale of English to check the CEFR level of any grammar, vocabulary, or text), including its rubrics (e.g. *”Arrange the words properly…” which is unusual wording and above level (‘arrange’ not B1), instead of e.g. “Put the words in the correct order to form sentences”)
· make sure that your options are real words, or grammatically accurate combinations of words (e.g. not *goed (for ‘went/gone’) or *gooder (for ‘better’)
· avoid using absolute words such as ‘all’, ‘never’, ‘always’, etc. (Similarly, avoid sweeping options such as ‘none of the above/all of the above’)
• your item doesn’t prompt only shallow processing by e.g. recycling the exact same image/sentence from the lesson/core content
• item is consistent with the scope of your TL (e.g. at the end of a lesson on the second conditional (taught as ‘if + past simple, would + infinitive’) instead of *’If I was going to meet a famous person, I would buy a pashmina’ use e.g. ‘If I met a famous person, I’d buy a pashmina’)
• avoid using grammatical terms in the rubric (e.g. *’Complete this past perfect sentence with the missing parts’ instead of e.g. ‘Complete the sentence with the missing words’)
· avoid using ‘a/an’ –where the options are preceded by the indefinite article, all options must either begin with a consonant and be preceded by the indefinite article ‘a’; or all options must begin with a vowel, and be preceded by the indefinite article ‘an’
P.s. and a reminder:
There IS life beyond just multiple choice (single response) questions! 😉 (…although other item types are often harder to create and therefore less common…).
Some other (also objectively marked) item types might include e.g. multiple choice multiple (!) response, select a blank (dropdown with 2-4 options), fill in the blanks, fill in the blanks on image, drag and drop, drag and drop on image, matching (3-6 items), etc.
…or in the case of IELTS reading:
diagram label completion, flow-chart completion, identifying information, identifying the writer’s views/claims, matching features, matching headings, matching information, matching sentence endings, multiple choice, note completion, sentence completion, short-answer questions, summary completion, table completion
…and on IELTS listening:
multiple choice, matching; plan, map, diagram labeling; form, note, table, flow-chart, summary, as well as sentence completion and short answer questions.
— —
Hi. I’m Fatime. I’m an IELTS Teacher Trainer, helping CELTA-qualified English language teachers become better at teaching SKILLS, as opposed to just testing them.
Check out my courses here:
How to Teach IELTS Listening:
How to Teach IELTS Reading:
How to Teach IELTS Writing:
How to Teach IELTS Speaking:
