Speech Disentanglement for Robust Custom Word Detection: Solution Architecture and Theoretical Foundations

This document serves as a comprehensive technical treatise on the mathematical, architectural, and optimization foundations of the DISENT-KWS system.

---

1. Problem Formulation and Theoretical Constraints

Let $x(t)$ be a monaural, continuous-time acoustic signal recorded in a noisy reverberant environment. The signal is modeled as:

$$x(t) = \left( s_T(t) * h_T(t) \right) + \sum_{i=1}^{I} \left( s_i(t) * h_i(t) \right) + n(t)$$

Where:

• $s_T(t)$ is the dry acoustic source signal of the target enrolled speaker.
• $s_i(t)$ represents the $i$-th interfering background speaker.
• $h_T(t)$ and $h_i(t)$ are the Room Impulse Responses (RIR) representing the acoustic paths from the respective sources to the microphone.
• $*$ denotes the mathematical convolution operator.
• $n(t)$ is additive environmental noise (e.g., babble, crowd, or traffic noise).

The objective is to design a mapping $f(x) \to D \in {0, 1}$ that satisfies:

$$D = 1 \iff \left( \mathcal{K}(x) = k_T \right) \land \left( \mathcal{S}(x) = s_T \right)$$

Where $\mathcal{K}(x)$ identifies the phonetic content (keyword), $k_T$ is the target custom word, $\mathcal{S}(x)$ identifies the speaker identity, and $s_T$ is the target enrolled speaker.

The mapping must satisfy the following joint operational constraints:

1. Parameter Budget: $\Theta(f) < 3.0 \times 10^6$ parameters.
2. Real-Time Factor (xRT): Let $\Delta \tau$ be the execution time of the model on a CPU for an audio segment of duration $T$. The real-time factor is bounded by: $$ \text{xRT} = \frac{\Delta \tau}{T} < 0.20 $$
3. Robustness: Keyword and speaker verification performance must generalize across signal-to-noise ratios $\text{SNR} \in [-5, 30]\text{ dB}$.
4. Latency: End-to-end inference latency $< 200\text{ ms}$ on a CPU for $2\text{s}$ audio windows.

---

2. Acoustic Front-End Feature Extraction

The raw digital waveform $x[n]$ sampled at $f_s = 16\text{ kHz}$ is framed using a sliding Hamming window $w[n]$ of duration $25\text{ ms}$ ($N = 400$ samples) with a hop size of $10\text{ ms}$ ($R = 160$ samples).

For each frame $t$, the Short-Time Fourier Transform (STFT) is defined as:

$$X(t, k) = \sum_{n=0}^{N-1} x[t R + n] \cdot w[n] \cdot e^{-j \frac{2 \pi k n}{N}}$$

The power spectrum $|X(t, k)|^2$ is mapped to a Mel-scale filterbank. The transformation from linear frequency $f$ to Mel frequency $m$ is defined by:

$$m = 2595 \cdot \log_{10}\left(1 + \frac{f}{700}\right)$$

Applying a bank of $M = 80$ triangular filters $H_m(k)$, we compute the Log Filter-Bank Energies (LFBE):

$$\text{LFBE}(t, m) = \log \left( \sum_{k=0}^{N/2} |X(t, k)|^2 \cdot H_m(k) + \epsilon \right)$$

Where $\epsilon = 10^{-6}$ is a stability term to prevent logarithmic divergence. The resulting representation for a $2$-second audio segment is a tensor $\mathbf{X} \in \mathbb{R}^{1 \times 80 \times 200}$.

---

3. Neural Architecture Design

3.1 Shared Encoder: Broadcasted Residual Network (BC-ResNet-2)

To extract robust local features while minimizing computation, we employ a Broadcasted Residual Network (BC-ResNet) variant.

Let $\mathbf{x} \in \mathbb{R}^{C_{in} \times F \times T}$ be the input to a Broadcasted Residual Block (BCResBlock). The block processes features along two pathways:

1. Frequency Broadcast Pathway: A 2D convolution operates on the frequency-time plane to capture local acoustic structures: $$ \mathbf{y}_{2D} = \text{ReLU}\left(\text{BatchNorm2D}\left(\text{Conv2D}(\mathbf{x}; \mathbf{W}_{2D})\right)\right) $$
2. Temporal Context Pathway: The frequency dimension is compressed via Average Pooling, and a 1D convolution captures temporal patterns: $$ \mathbf{y}_{1D} = \text{ReLU}\left(\text{BatchNorm1D}\left(\text{Conv1D}\left(\frac{1}{F}\sum_{f=1}^F \mathbf{x}[:, f, :]; \mathbf{W}_{1D}\right)\right)\right) $$

The temporal feature map $\mathbf{y}{1D} \in \mathbb{R}^{C{out} \times T}$ is broadcasted along the frequency dimension and combined with the 2D path:

$$ \mathbf{y}_{block} = \mathbf{y}_{2D} + \text{Broadcast}\left(\mathbf{y}_{1D}, \text{target\_shape}=(C_{out}, F', T)\right) $$

This mechanism allows the model to compute global temporal statistics while retaining spatial-frequency context using only $33.8\text{ K}$ parameters.

3.2 Phonetic Head: Causal Conformer

The phonetic head maps the intermediate representations to a content-discriminative embedding $\mathbf{z}_{phn} \in \mathbb{R}^{192}$. It comprises Causal Conformer blocks, which merge the global context modeling of Multi-Head Self-Attention (MHSA) with the local extraction of Convolutional modules.

A Conformer block contains four modules arranged in a Macaron-style structure:

$$\tilde{\mathbf{x}}_i = \mathbf{x}_{i-1} + \frac{1}{2} \text{FFN}(\mathbf{x}_{i-1})$$$$\mathbf{x}'_i = \tilde{\mathbf{x}}_i + \text{MHSA}(\tilde{\mathbf{x}}_i)$$$$\mathbf{x}''_i = \mathbf{x}'_i + \text{CausalConv1D}(\mathbf{x}'_i)$$$$\mathbf{x}_i = \text{LayerNorm}\left(\mathbf{x}''_i + \frac{1}{2} \text{FFN}(\mathbf{x}''_i)\right)$$

Causal Attention Constraint:

To prevent the model from looking at future frames during streaming real-time inference, the Multi-Head Self-Attention module employs a causal masking matrix $\mathbf{M} \in \mathbb{R}^{T \times T}$:

$$A_{i,j} = \frac{\mathbf{q}_i \mathbf{k}_j^T}{\sqrt{d_k}} + M_{i,j}$$

Where:

$$ M_{i,j} = \begin{cases} 0 & \text{if } j \le i \\ -\infty & \text{if } j > i \end{cases} $$

This ensures that the attention weights for future frames ($j > i$) resolve to exactly zero after the softmax operation.

3.3 Speaker Head: ECAPA-TDNN Lite

The speaker head maps the representations to a speaker-discriminative embedding $\mathbf{z}_{spk} \in \mathbb{R}^{192}$. It uses a 1D Squeeze-and-Excitation (SE) Dilated Conv block followed by Attentive Statistics Pooling (ASP).

The channel attention vector $\mathbf{s} \in \mathbb{R}^C$ in the SE block is calculated as:

$$\mathbf{s} = \sigma\left(\mathbf{W}_2 \cdot \text{ReLU}\left(\mathbf{W}_1 \cdot \left(\frac{1}{T}\sum_{t=1}^T \mathbf{h}_t\right)\right)\right)$$

Where $\sigma(\cdot)$ is the sigmoid function, and $\mathbf{h}_t$ represents the frame-level activations. The channel-wise scaling is defined by:

$$\tilde{\mathbf{h}}_t = \mathbf{s} \odot \mathbf{h}_t$$

Attentive Statistics Pooling (ASP):

Instead of calculating simple arithmetic means and standard deviations, ASP calculates temporal attention weights $\alpha_t$ to focus on speaker-distinct frames:

$$e_t = \mathbf{v}^T \tanh\left(\mathbf{W} \tilde{\mathbf{h}}_t + \mathbf{b}\right)$$$$\alpha_t = \frac{e^{e_t}}{\sum_{\tau=1}^T e^{e_{\tau}}}$$

The attention-weighted mean vector $\boldsymbol{\mu}$ and standard deviation vector $\boldsymbol{\sigma}$ are computed as:

$$\boldsymbol{\mu} = \sum_{t=1}^T \alpha_t \tilde{\mathbf{h}}_t$$$$\boldsymbol{\sigma} = \sqrt{\sum_{t=1}^T \alpha_t \left(\tilde{\mathbf{h}}_t - \boldsymbol{\mu}\right)^2}$$

The speaker embedding is then projected to the final dimension:

$$\mathbf{z}_{spk} = \mathbf{W}_{proj} \cdot [\boldsymbol{\mu}; \boldsymbol{\sigma}] + \mathbf{b}_{proj}$$

---

4. Theory of Speech Disentanglement

Standard keyword and speaker models trained jointly share acoustic traits, meaning phonetic representations often retain speaker identity information. To achieve clean disentanglement in the feature-space, we implement an adversarial constraint and a variational mutual information bound.

4.1 Gradient Reversal Layer (GRL)

Let $G_{back}$ be the shared encoder, $G_{phn}$ be the phonetic head, and $D_{spk}$ be an auxiliary adversarial speaker classifier. The network optimizes the classification loss $L_{kw}$ and an adversarial speaker classification loss $L_{adv}$:

$$E(G_{back}, G_{phn}, D_{spk}) = L_{kw}(G_{back}, G_{phn}) - \lambda L_{adv}(G_{back}, D_{spk})$$

To train this end-to-end using standard backpropagation, we define the GRL mapping $R_\lambda(\mathbf{x})$:

• Forward Propagation: $$ R_\lambda(\mathbf{x}) = \mathbf{x} $$
• Backward Propagation: $$ \frac{d R_\lambda(\mathbf{x})}{d\mathbf{x}} = -\lambda \mathbf{I} $$

During backward propagation, the gradients flowing from the adversarial speaker classifier are inverted and scaled by $-\lambda$, forcing the shared encoder to delete speaker identity markers from the features sent to the phonetic head.

4.2 Contrastive Log-ratio Upper Bound (CLUB) Mutual Information Minimizer

The Mutual Information (MI) between the phonetic representation $\mathbf{Z}{phn}$ and the speaker representation $\mathbf{Z}{spk}$ is defined as:

$$I(\mathbf{Z}_{phn}; \mathbf{Z}_{spk}) = \mathbb{E}_{P(\mathbf{Z}_{phn}, \mathbf{Z}_{spk})} \left[ \log \frac{P(\mathbf{Z}_{phn} | \mathbf{Z}_{spk})}{P(\mathbf{Z}_{phn})} \right]$$

Since the true conditional probability $P(\mathbf{Z}{phn} | \mathbf{Z}{spk})$ is intractable, we approximate it using a variational neural network estimator $q_\theta(\mathbf{z}{phn} | \mathbf{z}{spk})$. The CLUB upper bound estimator for a batch of size $B$ is formulated as:

$$I_{CLUB}(\mathbf{Z}_{phn}; \mathbf{Z}_{spk}) = \frac{1}{B} \sum_{i=1}^B \left[ \log q_\theta(\mathbf{z}_{phn, i} | \mathbf{z}_{spk, i}) - \frac{1}{B} \sum_{j=1}^B \log q_\theta(\mathbf{z}_{phn, j} | \mathbf{z}_{spk, i}) \right]$$

The optimization alternates between two steps:

1. Estimator Update: Maximize the variational log-likelihood to fit the conditional distribution: $$ \max_\theta \frac{1}{B} \sum_{i=1}^B \log q_\theta(\mathbf{z}_{phn, i} | \mathbf{z}_{spk, i}) $$
2. Feature Disentanglement: Minimize the CLUB mutual information bound with respect to the encoder parameters: $$ \min_{\Theta} I_{CLUB}(\mathbf{Z}_{phn}; \mathbf{Z}_{spk}) $$

This minimizer penalizes statistical dependencies between $\mathbf{z}{phn}$ and $\mathbf{z}{spk}$, forcing the embedding heads to represent independent acoustic attributes.

---

5. Loss Functions and Training Stages

The system is trained using a multi-loss optimization function across four stages: Phase 1 (softmax pre-training), Phase 2 (disentanglement & joint fine-tuning), Phase 3a (GE2E speaker refinement), and Phase 3b (hard-negative GE2E). The complete training pipeline is illustrated below:

5.1 Additive Angular Margin (AAM-Softmax) Loss

Used in Phase 1 to train classification boundaries. The loss function projects embeddings onto a hypersphere and introduces an angular margin $m$:

$$L_{AAM} = -\frac{1}{B} \sum_{i=1}^B \log \frac{e^{s \cdot \cos(\theta_{y_i} + m)}}{e^{s \cdot \cos(\theta_{y_i} + m)} + \sum_{j \neq y_i} e^{s \cdot \cos(\theta_j)}}$$

Where $\theta_j$ is the angle between the embedding vector and the class weight vector $\mathbf{W}_j$, $s$ is the scaling factor, and $m$ is the angular margin.

5.2 Prototypical Triplet and Rejection Loss

In Phase 2, we introduce triplet and rejection losses to reject unauthorized speakers and phonetically similar words. Let $\mathbf{z}_a$ be an anchor keyword embedding, $\mathbf{z}_p$ a positive (same keyword, same speaker) embedding, and $\mathbf{z}_n$ a negative (confuser keyword or unauthorized speaker) embedding.

We optimize the triplet loss:

$$L_{triplet} = \max\left(0, \|\mathbf{z}_a - \mathbf{z}_p\|_2^2 - \|\mathbf{z}_a - \mathbf{z}_n\|_2^2 + \alpha\right)$$

To handle out-of-vocabulary and unauthorized speakers, we define the rejection loss $L_{reject}$:

$$L_{reject} = \max\left(0, \gamma - \|\mathbf{z}_a - \mathbf{z}_{confuser}\|_2^2\right)$$

Where $\gamma$ is the rejection safety margin.

5.3 Generalized End-to-End (GE2E) Loss

In Phase 3a, the speaker head is refined using the GE2E loss. For a batch of $N$ speakers each with $M$ utterances, let $\mathbf{x}{ji}$ denote utterance $i$ from speaker $j$. The speaker embedding for utterance $i$ of speaker $j$ is $\mathbf{e}{ji} = f_{spk}(\mathbf{x}_{ji}) \in \mathbb{R}^{192}$. The centroid (mean embedding) for speaker $j$ excluding utterance $i$ is:

$$\mathbf{c}_{j}^{(-i)} = \frac{1}{M-1} \sum_{m \neq i} \mathbf{e}_{jm}$$

The similarity matrix $\mathbf{S}$ with entries $S_{ji,k} = w \cdot \cos(\mathbf{e}_{ji}, \mathbf{c}_k) + b$ (where $w, b$ are learnable scale/bias) is optimized via softmax:

$$L_{GE2E} = -\frac{1}{N \cdot M} \sum_{j=1}^N \sum_{i=1}^M \log \frac{e^{S_{ji,j}}}{\sum_{k=1}^N e^{S_{ji,k}}}$$

5.4 Hard-negative GE2E Loss

In Phase 3b, the GE2E loss is extended with hard-negative mining from LibriPhrase. For each anchor utterance, a phonetically similar confuser utterance from a different speaker is retrieved. The similarity of the anchor to this hard negative is explicitly penalized:

$$L_{hard} = L_{GE2E} + \beta \cdot \max\left(0, \cos(\mathbf{e}_{anchor}, \mathbf{e}_{hardneg}) - \delta\right)$$

Where $\beta$ is the hard-negative weight and $\delta$ is the margin.

---

6. Scorer Calibration and Verification

At evaluation time, the similarity of the test embedding against the enrolled reference prototypes is measured using cosine similarity:

$$\text{Sim}(\mathbf{z}_1, \mathbf{z}_2) = \frac{\mathbf{z}_1 \cdot \mathbf{z}_2^T}{\|\mathbf{z}_1\|_2 \|\mathbf{z}_2\|_2}$$

The Dual-Gate Scorer combines the similarities linearly:

$$\text{Score} = w_{kw} \cdot \text{Sim}(\mathbf{z}_{phn}, \mathbf{p}_{kw}) + w_{spk} \cdot \text{Sim}(\mathbf{z}_{spk}, \mathbf{p}_{spk})$$

Using grid search on joint verification trials, we optimized the parameters to:

$$w_{kw} = 0.30, \quad w_{spk} = 0.65, \quad \tau_{EER} = 0.2222$$

For streaming verification, the score is smoothed over time using an Exponential Moving Average (EMA) with coefficient $\alpha_{ema} = 0.7$:

$$\bar{S}_t = \alpha_{ema} \cdot \bar{S}_{t-1} + (1 - \alpha_{ema}) \cdot \text{Score}_t$$

The output trigger decision is evaluated as:

$$D = \begin{cases} 1 & \text{if } \bar{S}_t \ge \tau_{EER} \\ 0 & \text{if } \bar{S}_t < \tau_{EER} \end{cases}$$

---

7. Experimental Results and Ablation Study

To evaluate the effectiveness of the disentangled representation learning and the dynamic scorer, we benchmarked the DISENT-KWS model on the test sets of Google Speech Commands v2 (11,005 samples) and VoxCeleb1 (1,251 speakers). The optimal scorer calibration was obtained via exhaustive grid search over $10 \times 10$ weight combinations, yielding $w_{kw}=0.30$, $w_{spk}=0.65$, and $\tau_{EER}=0.2222$.

7.1 Quantitative Benchmark Results

Metric	Value	Notes
Parameters	1.806 M	Within the $< 3.0$ M budget ✅
ONNX Model Size	0.60 MB	INT8 quantized export
CPU Latency	26.43 ms	p95: 28.29 ms, $< 200$ ms target ✅
Real-Time Factor (xRT)	0.0132	Well under $< 0.20$ budget ✅
Keyword EER (standalone)	4.69%	Evaluated on 35-class GSC v2 test set (11,005 samples)
Speaker EER (standalone)	17.86%	Evaluated on VoxCeleb1 (1,251 speakers)
Joint EER	23.47%	Combined keyword + speaker verification trials
Joint AUC	0.8425	Area under the joint DET curve
Optimal Weights	wₖw=0.30, wₛₚₖ=0.65	Grid-searched over $10 \times 10$ combinations
EER Threshold (τ)	0.2222	Operating point at equal error rate

The Detection Error Trade-off (DET) curve illustrating the False Reject Rate (FRR) against the False Acceptance Rate (FAR) under joint keyword and speaker verification trials is shown below:

The parameter budget allocation across model components is visualized below:

7.2 Ablation Study

To measure the individual contributions of each architectural component, we systematically disabled modules and re-evaluated on the same test configuration:

Configuration	Keyword EER (%)	Speaker EER (%)	Parameters	Observations
Full Model (baseline)	4.69	17.33	1.806 M	Complete system with all components enabled.
No FiLM Conditioning	4.69	17.33	1.683 M	FiLM removal saves 123K parameters but does not affect EER on this test set.
No Speaker Head	4.69	N/A	1.806 M	KWS-only mode; speaker verification is entirely disabled.
No Temporal Block	11.22	25.48	1.796 M	Temporal context removal degrades both keyword (+6.53 pp) and speaker (+8.15 pp) EER significantly.
Equal Scorer Weights	4.69	17.33	1.806 M	Using $w_{kw}=0.50$, $w_{spk}=0.50$ instead of calibrated 0.30/0.65 weights.

The ablation study results are also visualized in the chart below:

7.3 SNR Robustness Evaluation

To validate performance across varying acoustic conditions, we evaluated the system at signal-to-noise ratios from -5 dB to 30 dB using MUSAN noise (babble, music, environmental). The system maintains consistent keyword detection accuracy across the full SNR range, demonstrating the effectiveness of our multi-condition training strategy:

---

8. Key Literature References

1. Broadcasted Residual Networks: Kim, B. et al. (2021). BC-ResNet: Broadcasted Residual Learning for Lightweight Noise-Robust Keyword Spotting. In Proc. Interspeech.
2. Conformer Architectures: Gulati, A. et al. (2020). Conformer: Convolution-augmented Transformer for Speech Recognition. In Proc. Interspeech.
3. ECAPA-TDNN: Desplanques, B. et al. (2020). ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. In Proc. Interspeech.
4. Variational Mutual Information Bounds: Cheng, P. et al. (2020). CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information. In Proc. ICML.
5. AAM-Softmax: Deng, J. et al. (2019). ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In Proc. CVPR.
6. SpeechBrain Toolkit: Ravanelli, M. et al. (2021). SpeechBrain: A Multi-Task Speech Toolkit. arXiv preprint arXiv:2106.04624.