Performance Evaluation
To evaluate a VAD, its output using test recordings is compared with those of an “ideal” VAD – created by hand-annotating the presence/absence of voice in the recordings. The performance of a VAD is commonly evaluated on the basis of the following four parameters:
- FEC (Front End Clipping): clipping introduced in passing from noise to speech activity;
- MSC (Mid Speech Clipping): clipping due to speech misclassified as noise;
- OVER: noise interpreted as speech due to the VAD flag remaining active in passing from speech activity to noise;
- NDS (Noise Detected as Speech): noise interpreted as speech within a silence period.
Although the method described above provides useful objective information concerning the performance of a VAD, it is only an approximate measure of the subjective effect. For example, the effects of speech signal clipping can at times be hidden by the presence of background noise, depending on the model chosen for the comfort noise synthesis, so some of the clipping measured with objective tests is in reality not audible. It is therefore important to carry out subjective tests on VADs, the main aim of which is to ensure that the clipping perceived is acceptable. This kind of test requires a certain number of listeners to judge recordings containing the processing results of the VADs being tested. The listeners have to give marks on the following features:
- Quality;
- Comprehension difficulty;
- Audibility of clipping.
These marks, obtained by listening to several speech sequences, are then used to calculate average results for each of the features listed above, thus providing a global estimate of the behavior of the VAD being tested. To conclude, whereas objective methods are very useful in an initial stage to evaluate the quality of a VAD, subjective methods are more significant. As, however, they are more expensive (since they require the participation of a certain number of people for a few days), they are generally only used when a proposal is about to be standardized.
Read more about this topic: Voice Activity Detection
Famous quotes containing the words performance and/or evaluation:
“Nobody can misunderstand a boy like his own mother.... Mothers at present can bring children into the world, but this performance is apt to mark the end of their capacities. They can’t even attend to the elementary animal requirements of their offspring. It is quite surprising how many children survive in spite of their mothers.”
—Norman Douglas (1868–1952)
“Evaluation is creation: hear it, you creators! Evaluating is itself the most valuable treasure of all that we value. It is only through evaluation that value exists: and without evaluation the nut of existence would be hollow. Hear it, you creators!”
—Friedrich Nietzsche (1844–1900)