Voice Activity Detection - Performance Evaluation

Performance Evaluation

To evaluate a VAD, its output using test recordings is compared with those of an “ideal” VAD – created by hand-annotating the presence/absence of voice in the recordings. The performance of a VAD is commonly evaluated on the basis of the following four parameters:

• FEC (Front End Clipping): clipping introduced in passing from noise to speech activity;
• MSC (Mid Speech Clipping): clipping due to speech misclassified as noise;
• OVER: noise interpreted as speech due to the VAD flag remaining active in passing from speech activity to noise;
• NDS (Noise Detected as Speech): noise interpreted as speech within a silence period.

Although the method described above provides useful objective information concerning the performance of a VAD, it is only an approximate measure of the subjective effect. For example, the effects of speech signal clipping can at times be hidden by the presence of background noise, depending on the model chosen for the comfort noise synthesis, so some of the clipping measured with objective tests is in reality not audible. It is therefore important to carry out subjective tests on VADs, the main aim of which is to ensure that the clipping perceived is acceptable. This kind of test requires a certain number of listeners to judge recordings containing the processing results of the VADs being tested. The listeners have to give marks on the following features:

• Quality;
• Comprehension difficulty;
• Audibility of clipping.

These marks, obtained by listening to several speech sequences, are then used to calculate average results for each of the features listed above, thus providing a global estimate of the behavior of the VAD being tested. To conclude, whereas objective methods are very useful in an initial stage to evaluate the quality of a VAD, subjective methods are more significant. As, however, they are more expensive (since they require the participation of a certain number of people for a few days), they are generally only used when a proposal is about to be standardized.