Skip to main content

Test Methods

Overview​

The QExE tool currently features five quality evaluation methods. To operate a test using the QExE tool, the methodType entry in the configuration file is a mandatory setting.

Four of these methods are traditionally used in audio quality evaluation settings, and is indicated using the modalityRatio : A flag. The remaining method can be used for audio, visual, or audiovisual quality evaluation and is indicated via the modalityRatio : A, V, or AV flag.

  • Multiple stimulus with hidden reference and anchor (MUSHRA) 
    "methodType" : "MUSHRA",
    "modalityRatio" : "A",
  • Multiple stimulus comparison (MS) 
    "methodType" : "MS",
    "modalityRatio" : "A",
  • Elimination-by-aspects (EBA) 
    "methodType" : "EBA",
    "modalityRatio" : "A",
  • Pairwise comparison (PC) 
    "methodType" : "PC",
    "modalityRatio" : "A",
  • Absolute catergory rating (ACR) 
    "methodType" : "ACR",
    "modalityRatio" : "A" / "V" / "AV",

As some of these names imply, the four audio quality evaluation methods employ multiple audio conditions in parallel, these being MUSHRA, MS, EBA, and PC. I.e., subjects are provided with two more buttons displayed as A, B, and so on. These represent the conditions subjects are intended to evaluate, and by clicking the condition buttons, the audio rendering will change depending on the condition employed for A or B.

The remaining method ACR is sometimes referred to as a "single stimulus" method, as only one condition is presented to the subject at a time. As there is no need to real-time switching between conditions, this method is commonly employed in visual quality evaluation studies, and thus provides a practical means for evaluation visual quality in virtual reality.

Method → Item Calculation​

The choice of which method you choose for your study, will impact the number of evaluation items are presented to the subject.

For example, a test employing ACR will present as many evaluation items, as there are stimuli x conditions. As ACR presents items individually, every combination will presented sequentially. In contrast, the MS method will present all conditions in parallel, per stimuli. So for each stimuli (i.e., VR scene) all three different conditions will be presented at the same time for the subject to switch between and evaluate at their discretion.

methodType :​

MUSHRA​

Multiple Stimulus with Hidden Reference and Anchor

The interface features a continuous scale from 0 - 100, with verbal labels 'Bad, Poor, Fair, Good, and Excellent'. The REF button will switch to the open referece. Conditions A, B, C ... etc., will switch between conditions. Users should set the slider to indicate the magnitude of a perceieved sentation (e.g., quality).

Your Image

MS​

Multiple Stimulus

The interface features a continuous scale from 0 - 100, with verbal labels 'Bad, Poor, Fair, Good, and Excellent'. No reference button is present making all ratings relative to one another. As with MUSHRA, conditions can be selected with buttons A, B, C, etc.

Your Image

EBA​

Elimination-by-Aspects

Sometimes referred to as rank-order or round-robin, the method features buttons to select a condition to play (A, B, C, etc.) along with buttons to eliminate the respective condition.

Your Image

PC​

Pairwise Comparison

The pariwise comparison method typically features a pair of conditions A and B. The method is often refered to as AB-testing. In this interface, the additional slider allows for a magnitude of the perferred choice to be expressed. Subject will then itterate through all condition pairs before moving onto the next text item.

Your Image

ACR​

Absolute Category Rating

Sometimes refered to as a single-stimlus method, the ACR method present only a single condition at a time. This means that all conditions are technically judged in isolation. In practice, factors such as acoustic or visual memory, leading-stimlus bias, and exposre bias will factor into subjects' perception. The rating applied is given along a 11-point absolute caegatory rating scale.

Your Image