Abstract
Suppose we have two tables of counts, each indexed by the same set of categories, and we wish to determine whether the underlying generating mechanism behind each table might be different. Furthermore, if the mechanism is different, we suspect it likely varies only in a small fraction of the observed categories. This question arises in several applications, including attributing authorship based on word frequencies.

We propose a tool for this problem built on P-values derived from a Binomial Allocation model and the notion of Higher Criticism (Donoho & Jin 2004). Our proposal offers an interpretable and easy-to-apply tool, and our theoretical analysis shows that it is powerful against the aforementioned changes in the generating mechanism. Specifically, under a calibration of the number of categories to rarity and signal intensity parameters, the power of our test experiences a phase transition that matches the phase transition of the likelihood ratio test.

Our analytic framework goes beyond contingency tables, encompassing a wide range of rare and weak signal models experiencing departures on a moderate scale. We discuss several interesting new models falling under this category, including the detection of a few edits within text written by a generative language model.