Test Methods
Overview
The QExE tool currently features five quality evaluation methods. The methodType entry in the configuration file is a mandatory setting for all tests.
Four methods are designed for audio quality evaluation and require the modalityRatio : "A" flag. The fifth — ACR — supports audio, visual, and audiovisual quality evaluation, and accepts "A", "V", or "AV" as the modality flag.
| Method | Full Name | Modality | Stimulus Presentation |
|---|---|---|---|
| MUSHRA | Multiple Stimulus with Hidden Reference and Anchor | Audio | Multiple, simultaneous |
| MS | Multiple Stimulus | Audio | Multiple, simultaneous |
| EBA | Elimination-by-Aspects | Audio | Multiple, simultaneous |
| PC | Pairwise Comparison | Audio | Pairwise, simultaneous |
| ACR | Absolute Category Rating | Audio / Visual / Audiovisual | Single |
MUSHRA, MS, EBA, and PC are multiple-stimulus methods: subjects are presented with two or more conditions simultaneously, selectable via buttons labelled A, B, and so on, with the audio rendering switching in real time. ACR is a single-stimulus method in which each condition is presented individually and rated in isolation.
Method → Item Calculation
The choice of method directly affects the number of evaluation items presented to each participant.
With ACR, every stimulus x condition combination is presented as a separate item and rated sequentially. With a multiple-stimulus method such as MS, all conditions for a given stimulus are presented in parallel within a single item — subjects switch between them freely before providing a rating. This distinction should be considered when balancing test duration against the number of conditions and stimuli.
methodType : MUSHRA
MUSHRA
The interface features a continuous scale from 0–100, with verbal labels Bad, Poor, Fair, Good, and Excellent. The REF button switches to the open reference. Condition buttons A, B, C, etc. switch between conditions. Subjects set the slider to indicate the magnitude of a perceived sensation (e.g., quality).
methodType : MS
Multiple-Stimulus
The interface features a continuous scale from 0–100, with verbal labels Bad, Poor, Fair, Good, and Excellent. No reference button is present, making all ratings relative to one another. As with MUSHRA, conditions can be selected with buttons A, B, C, etc.
methodType : EBA
Elimination-by-aspects
Sometimes referred to as rank-order or round-robin, this method features buttons to select a condition to play (A, B, C, etc.) alongside buttons to eliminate the respective condition from consideration.
methodType : PC
Pairwise comparison
The pairwise comparison method presents a pair of conditions A and B — often referred to as AB-testing. An additional slider allows the magnitude of preference for the chosen condition to be expressed. Subjects iterate through all condition pairs before progressing to the next test item.
methodType : ACR
Absolute category rating
A single-stimulus method, ACR presents one condition at a time, meaning all conditions are technically judged in isolation. In practice, factors such as acoustic or visual memory, leading-stimulus bias, and exposure bias will influence subjects' ratings. Ratings are given on an 11-point absolute category rating scale.