General Frameworks for Conditional Two-Sample Testing

arXiv stat.ML / 5/5/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper addresses conditional two-sample testing: deciding whether two populations share the same distribution after controlling for confounding factors, a task common in domain adaptation and algorithmic fairness.
It proves a hardness result showing that any valid test cannot achieve nontrivial power against an arbitrary single alternative unless additional assumptions are imposed.
The authors propose two general frameworks for achieving validity and power by targeting specific classes of distributions: one reduces the task to conditional independence testing, and the other reduces it to marginal two-sample testing via estimated density ratios.
They provide concrete implementations using classification and kernel-based methods, and validate the approaches through simulation studies under finite-sample settings.

Abstract

We study the problem of conditional two-sample testing, which aims to determine whether two populations have the same distribution after accounting for confounding factors. This problem commonly arises in various applications, such as domain adaptation and algorithmic fairness, where comparing two groups is essential while controlling for confounding variables. We begin by establishing a hardness result for conditional two-sample testing, demonstrating that no valid test can have significant power against any single alternative without proper assumptions. We then introduce two general frameworks that implicitly or explicitly target specific classes of distributions for their validity and power. Our first framework allows us to convert any conditional independence test into a conditional two-sample test in a black-box manner, while preserving the asymptotic properties of the original conditional independence test. The second framework transforms the problem into comparing marginal distributions with estimated density ratios, which allows us to leverage existing methods for marginal two-sample testing. We demonstrate this idea in a concrete manner with classification and kernel-based methods. Finally, simulation studies are conducted to illustrate the proposed frameworks in finite-sample scenarios.