New framework for auditing machine unlearning

Machine unlearning must be auditable, but conventional two-sample tests can be expensive, insensitive to localized changes, and prone to false positives. Google Research introduces Regularized f-Divergence Kernel Tests, which compare an unlearned model relative to both a safely retrained model and the original compromised model. The framework adaptively detects meaningful privacy or unlearning failures while controlling false positives and improving sensitivity as more samples become available.

Why Standard Auditing Methods Fall Short

• Auditors often lack access to model internals or the original training data, so they must assess unlearning by querying models and comparing outputs.
• Standard two-sample tests require many samples to distinguish genuine violations from random variation in large models.
• Maximum mean discrepancy (MMD) is effective for broad distribution shifts but can miss:
  • Rare, highly localized outliers
  • Non-smooth differences
  • Changes that require careful kernel-bandwidth tuning
• Existing methods often require manual selection of statistics and hyperparameters.
• Comparing an unlearned model only with a retrained reference can produce false positives because independently trained models may differ due to factors such as batch size.
• Perfect retrain equivalence is generally impossible for local unlearning methods, since they retain traces of the original training process.

Regularized f-Divergence Kernel Tests

• The framework uses a relative-distance test to determine whether an unlearned model is closer to:
  • A safely retrained model that never saw the data
  • The original model that still contains the data
• It supports multiple f-divergences for different types of distribution changes:
  • Chi-squared and KL divergence: Useful for smooth or localized differences, including outliers.
  • Hockey-stick divergence: Designed for privacy and unlearning thresholds, ignoring differences below a defined safety budget while flagging meaningful breaches.
• Kernel regularization makes high-dimensional divergence estimation computationally tractable.
• The adaptive procedure selects suitable divergences and hyperparameters automatically.
• It avoids the need for sample splitting.
• The authors theoretically show that false-positive rates are controlled for any sample size, while false-negative risk approaches zero as the number of samples increases.

Experimental Evaluation

• The framework was tested on:
  • Synthetic perturbed-uniform distributions
  • The Expo1D outlier-detection task from high-energy physics
• High-energy physics provides a stringent benchmark because detecting rare particles requires identifying extremely subtle distributional differences.
• For differential-privacy auditing, the researchers compared outputs generated from datasets differing by one record:
  • A private mechanism should produce statistically indistinguishable samples.
  • A non-private mechanism should reveal a detectable difference.
• For machine-unlearning evaluation, the researchers used a three-sample relative test rather than simply comparing an unlearned model with a gold-standard retrained model.
• The evaluation included methods such as Selective Synaptic Dampening, pruning, and random-label techniques.

Practical Recommendation

Auditors should use relative, divergence-based testing rather than relying on a single comparison with a retrained reference model. Regularized f-divergence kernel tests offer a more flexible way to detect localized privacy leaks and unlearning failures while reducing false alarms and costly manual tuning.