{"ID":2888061,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.00770","arxiv_id":"2508.00770","title":"On admissibility in post-hoc hypothesis testing","abstract":"The validity of classical hypothesis testing requires the significance level $α$ be fixed before any statistical analysis takes place. This is a stringent requirement. For instance, it prohibits updating $α$ during (or after) an experiment due to changing concern about the cost of false positives, or to reflect unexpectedly strong evidence against the null. Perhaps most disturbingly, witnessing a p-value $p\\llα$ vs $p= α- ε$ for tiny $ε\u003e 0$ has no (statistical) relevance for any downstream decision-making. Following recent work of Grünwald (2024), we develop a theory of post-hoc hypothesis testing, enabling $α$ to be chosen after seeing and analyzing the data. To study \"good\" post-hoc tests we introduce $Γ$-admissibility, where $Γ$ is a set of adversaries which map the data to a significance level. We classify the set of $Γ$-admissible rules for various sets $Γ$, showing they must be based on e-values, and recover the Neyman-Pearson lemma when $Γ$ is the constant map.","short_abstract":"The validity of classical hypothesis testing requires the significance level $α$ be fixed before any statistical analysis takes place. This is a stringent requirement. For instance, it prohibits updating $α$ during (or after) an experiment due to changing concern about the cost of false positives, or to reflect unexpec...","url_abs":"https://arxiv.org/abs/2508.00770","url_pdf":"https://arxiv.org/pdf/2508.00770v3","authors":"[\"Ben Chugg\",\"Tyron Lardy\",\"Aaditya Ramdas\",\"Peter Grünwald\"]","published":"2025-08-01T16:45:58Z","proceeding":"math.ST","tasks":"[\"math.ST\",\"stat.ME\"]","methods":"[]","has_code":false}