About vad

SG-VAD is a speech activity detection (VAD) model based on stochastic gates, implementing an ICASSP 2023 research paper from Jonathan Svirsky and Ofir Lindenbaum. The system uses a mask or filter architecture where noise audio and spoken words are processed as separate categories during training, then combined into a unified VAD model for inference. Built on NVIDIA's NeMo framework, it supports custom label counts and flexible training via manifest files. The repository includes a pre-trained PyTorch checkpoint (sgvad.pth) and an inference script that applies a configurable threshold on model output to classify speech versus non-speech. On the AVA-speech test set, the published checkpoint achieves an EER of 10.40%, TPR at FPR 0.315 of 0.96, and ROCAUC of 0.95. After fixing a label creation bug for the HAVIC benchmark and restricting non-speech categories to noise, background noise, music, and baby, EER improved to 21.33% with ROCAUC of 85.31. The original HAVIC results show EER of 23.29%, TPR at FPR 0.315 of

j

Published by

jsvir

Visit View Profile

README.md

View on GitHub

An official implementation of the ICASSP 2023 paper: SG-VAD: Stochastic Gates Based Speech Activity Detection

UPDATES:

2025-03-24: we have found a bug in label creation for HAVIC test set, after fixing it and including only the categories ['noise', 'background_noise', 'music', 'baby'] as non-speech classes, we get better than reported in the paper results:

HAVIC test
- EER: 21.33%
- ROCAUC: 85.31
2024-01-03: removed hard-coded dependency on 36 labels exactly. Now you can define any number of labels you want and the code should support it.

Evaluation results (published checkpoint)

AVA-speech test

EER=10.40%
TPR@FPR=0.315 is 0.96
ROCAUC=0.95
HAVIC test
EER=23.29%
TPR@FPR=0.315 is 0.91
ROCAUC=0.83

Training

To train a new model you need to prepare the next dataset:

The manifest file with noise audio file paths (the same duration as the audios with words), the label should be "background"
The manifest file with spoken words file paths, each word should be labeled with one category

Please note that, as described in the paper, the main VAD model is trained as mask/filter as presented in the next schema:

Once the training is finished, the final model architecture is:

Prepare your dataset in manifest format supported by NeMo
Update config file with your paths and hyper-params
Install NeMo requirements
Run train.py script.

Inference:

We publish a pre-trained Pytorch checkpoint (sgvad.pth)
To use the published checkpoint as-is you need to calibrate the threshold for model output. All values under the threshold are predicted as Non-speech.
The default value for the threshold is 3.5, but it may be too aggressive for your application.
To try it on test audios run python sgvad.py

Aknowledgements:

We thank NeMo team for their great open-source repo.

Citation:

@inproceedings{svirsky2023sg,
  title={SG-VAD: Stochastic Gates Based Speech Activity Detection},
  author={Svirsky, Jonathan and Lindenbaum, Ofir},
  booktitle={ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={1--5},
  year={2023},
  organization={IEEE}
}

vad