For high stakes tests or exams, in particular in higher education, teachers want to analyse the quality of their questions. There are established methods for this, looking at the difficulty and discrimination of each question based on the student answers. This is not something that we have had in itslearning before, and we are happy to announce that such an analysis is now available. The option is found in the Results tab, close to the summary information for the test. Simply press "Export question analasys". This triggers the calculation and subsequent download of an excel (.xsls) file. The analysis is performed on all counting attempts that are fully assessed (max one per student). If a student later submits a new attempt that becomes the counting attempt, simply click the button again to trigger a new analysis.
The excel spreadsheet will have a header row, followed by a row with a short descriptions of what each column contains. There is one row for each question in the test.
Further column classification:
Explanations for each columnColumnPossible valuesExplanationColumnPossible valuesExplanation
Question number Integers Question number as seen in the question list in the test by the teacher. Question text Up to 100 characters of text
Question text, cut at 100 characters if long. HTML is stripped.
For Fill in the blank or Select from a list, this is the introduction text, not the question.
Question type Text Name of the question type (currently 7 available) Assessed and counting answers Integers Number of times the questions has been used in a fully assessed, counting attempt. Max possible score Positive number, 2 decimals Question score as set by the teacher when creating the question. Usually an integer, but decimal values are allowed. Average score Positive number, 2 decimals Average score achieved in the attempts included in the analysis. Standard deviation Positive number, 2 decimals Standard deviation of the scores achieved in the attempts included in the analysis. Calculated as standard deviation for a sample population. Variance Positive number, 2 decimals Variance of the scores achieved in the attempts included in the analysis. Calculated as variance for a sample population. Answers with max score (%) Positive number, 2 decimals Percentage of attempts with score = max score, based on the number of attempts included in the analysis. Difficulty Hard, medium or easy
The difficulty is based on the percentage of attempts which achieved the max score, and categorised as follows:
> 80% = Easy (meaning more than 80% of students got this correct)
30-80% = Medium
< 30% = Hard (meaning less than 30% of students got this correct)
Discrimination score Positive number between 0 and 1, 2 decimals, or empty.
The discrimination score is calculated as Pearson's correlation coefficiant for a sample population. The coefficient cannot be calculated in cases where all attempts have the same score.
The formula for this is
Good, fair, poor or cannot calculate
The discrimination is based on the discrimination score, and categorised as follows:
> 0.3 = Good
0.1 - 0.3 = Fair
< 0.1 = Poor
Review recommended Yes or no
Based on the difficulty and discrimination values, we will recommend review of the question if at least one of the following conditions is met:
Difficulty = Easy
Difficulty = Hard
Discrimination = Poor