Swarnim Tripathi

Identity Is Not the Keyword

Wed, 01 Jul 2026 23:05:53 +0530

A deep dive into DISENT-KWS: a 1.8M parameter model that disentangles phonetic content from speaker identity for robust custom keyword spotting on edge devices. Combines adversarial training, mutual information minimization, and dual-head scoring for speaker-aware wake word detection.

Deconstructing my PyTorch ATen PR: What happens when you index a Tensor?

Sun, 17 May 2026 01:00:00 +0530

My PyTorch PR that fixed empty-index crashes in ATen's tensor indexing... tracing the dispatch path from Python to C++ and the validation fix in IndexingUtils.h.

Mon, 01 Jan 0001 00:00:00 +0000

Model Card: DISENT-KWS

Model Description

DISENT-KWS is a speech disentanglement model for robust custom word detection. It uses a shared BC-ResNet-2 encoder with dual heads — a Causal Conformer (phonetic) and ECAPA-TDNN Lite (speaker) — to produce orthogonal latent embeddings. A Dual-Gate Scorer combines keyword and speaker similarity with calibrated weights for joint verification.

Total Parameters: 1.806M
ONNX Size: 0.60 MB (INT8 quantized)
Input: 80-band log Mel filterbank energies, 200 frames (2 s @ 16 kHz)
Output: z_phn ∈ ℝ¹⁹² (phonetic embedding), z_spk ∈ ℝ¹⁹² (speaker embedding)

Intended Use

Custom keyword spotting with speaker verification for access control, voice assistants, or wake-word systems
Real-time streaming inference on CPU (26.4 ms per 2 s window)
Few-shot enrollment (5+ utterances) for new users and custom words

Architecture

Component	Type	Params
Shared Encoder	BC-ResNet-2	33.8K
Temporal Block	Mamba SSM / Dilated Conv1D	10.3K
Phonetic Head	Causal Conformer (4 heads, kernel 15)	1,673K
Speaker Head	ECAPA-TDNN Lite (SE ratio 4, scale 4)	88.8K
Scorer	Dual-Gate (w_kw=0.30, w_spk=0.65, EMA α=0.7)	—

Training Data

Dataset	Usage	Samples
Google Speech Commands v2	Keyword pre-training (35 classes)	105K
VoxCeleb1	Speaker verification (1,251 speakers)	153K
LibriPhrase	Hard-negative triplet pairs	3K triplets
MUSAN	Noise augmentation (babble, music, environmental)	109 hrs

Training Phases

Phase 1: AAM-Softmax pre-training on keyword (GSC) and speaker (VoxCeleb) separately
Phase 2: Joint training with GRL adversarial reversal + CLUB MI minimization + triplet rejection loss
Phase 3a: GE2E speaker head refinement
Phase 3b: Hard-negative GE2E with LibriPhrase confusers

Performance

Metric	Value
Keyword EER	4.69%
Speaker EER	17.86%
Joint EER	23.47%
Joint AUC	0.8425
CPU Latency	26.43 ms (p95: 28.29 ms)
Real-Time Factor (xRT)	0.0132
Optimal Threshold (τ)	0.2222

Hardware & Runtime

Training: NVIDIA GPU with 16 GB+ VRAM (tested on Tesla T4)
Inference: CPU-only via ONNX Runtime; no GPU required
Memory: ~200 MB RAM at runtime

Known Limitations

Requires clean enrollment recordings (SNR ≥ 10 dB recommended)
Speaker EER (17.86%) is higher than keyword EER — joint verification mitigates this
Mamba SSM fallback to Dilated Conv1D on platforms without CUDA (no performance loss)

License

MIT

Mon, 01 Jan 0001 00:00:00 +0000

DISENT-KWS — Speech Disentanglement for Robust Custom Word Detection

Demo Video Link — YouTube Demo Video
Setup & Result Reproducibility Video Link — YouTube Setup Video

Quick Start

# Install dependencies
uv sync --all-extras

# Run tests (verify 60+ tests pass)
make test

# Option A: Live record 5 utterances from mic, then enroll
python src/demo.py record --model model_final.pt --out enrollment.pt

# Option B: Provide pre-recorded WAV files
python src/demo.py enroll \
 --recordings ./my_recordings/*.wav \
 --model model_final.pt \
 --out enrollment.pt

# Real-time detection
python src/demo.py detect --enrollment enrollment.pt --auto-threshold --vad-threshold 0.02

Project Artifacts

Technical Documentation

All technical documentation is organized in the docs/ directory:

Mon, 01 Jan 0001 00:00:00 +0000

Speech Disentanglement for Robust Custom Word Detection: Solution Architecture and Theoretical Foundations

This document serves as a comprehensive technical treatise on the mathematical, architectural, and optimization foundations of the DISENT-KWS system.

1. Problem Formulation and Theoretical Constraints

Let $x(t)$ be a monaural, continuous-time acoustic signal recorded in a noisy reverberant environment. The signal is modeled as:

$$x(t) = \left( s_T(t) * h_T(t) \right) + \sum_{i=1}^{I} \left( s_i(t) * h_i(t) \right) + n(t)$$

Where:

Mon, 01 Jan 0001 00:00:00 +0000

Testing Guide for DISENT-KWS

Overview

This project uses pytest for unit testing, GitHub Actions for CI-CD, and pre-commit hooks for local code quality enforcement.

All dependencies are managed via uv (fast Python package installer). Make sure uv is installed before running tests.

Quick Start

Run All Tests Locally

# Option 1: Using make
make test

# Option 2: Using uv directly
uv run pytest tests/ -v

# Option 3: Using the Python runner
python scripts/test.py

Run Tests with Coverage

# Using make
make test-cov

# Using uv directly
uv run pytest tests/ -v --cov=data --cov=models --cov=training --cov-report=html

# Using the Python runner
python scripts/test.py --coverage

Run Specific Tests

# Run only dataloader tests
make test-dataloaders
# or
uv run pytest tests/test_dataloaders.py -v

# Run specific test by name
uv run pytest tests/test_dataloaders.py::TestLFBETransform::test_transform_1d_input -v

# Run tests matching a pattern
uv run pytest tests/ -k "lfbe" -v

Test Structure

Test Files

`tests/test_dataloaders.py`

Tests for data loading and feature extraction: