{"ID":6536479,"CreatedAt":"2026-07-14T01:21:01.169441415Z","UpdatedAt":"2026-07-14T16:42:43.195211275Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.10391","arxiv_id":"2607.10391","title":"Vertical Fusion: Condensing Internal Representations for Robust ViT Classification","abstract":"Despite exposing rich intermediate representations, Vision Transformers (ViTs) are almost exclusively utilized as black-box feature extractors, where only the last layer is considered for downstream tasks. We challenge this convention by introducing the notion of recoverability: the capacity of intermediate representations to correct last-layer failures. By evaluating independent classification probes at every model depth across 16 datasets, we observe that intermediate probes correctly classify 18% to 76% of samples that the last-layer probe misclassifies. We show that these gains are not primarily driven by predictive diversity, but by a redundancy-correctness correspondence, where the internal hierarchy acts as a series of stable, redundant probes of a shared discriminative signal. While established horizontal ensemble strategies (i.e., across multiple models) can improve performance, they incur high computational cost and ignore this vertical signal within a single model. To bridge this gap, we propose VFusion, a principled vertical aggregation strategy employing a learnable mapping into a low-dimensional latent space that synthesizes features across the internal ViT hierarchy. VFusion substantially outperforms established aggregation baselines in both in-distribution and out-of-distribution settings, notably closing 45% of the accuracy gap between the best individual layer and a theoretical oracle performance. Our gains consistently generalize across model sizes and pre-training regimes, confirming that VFusion offers a robust and efficient alternative to horizontal ensemble methods. The code is available at https://github.com/francescodisalvo05/vit-vertical-fusion.","short_abstract":"Despite exposing rich intermediate representations, Vision Transformers (ViTs) are almost exclusively utilized as black-box feature extractors, where only the last layer is considered for downstream tasks. We challenge this convention by introducing the notion of recoverability: the capacity of intermediate representat...","url_abs":"https://arxiv.org/abs/2607.10391","url_pdf":"https://arxiv.org/pdf/2607.10391v1","authors":"[\"Francesco Di Salvo\",\"Shyam Nandan Rai\",\"Hamed Damirchi\",\"Ignacio Meza De la Jara\",\"Sebastian Doerrich\",\"Marco Lents\",\"Christian Ledig\"]","published":"2026-07-11T16:44:25Z","proceeding":"cs.CV","tasks":"[\"cs.CV\",\"cs.LG\",\"eess.IV\"]","methods":"[\"Vision Transformer\",\"Transformer\"]","has_code":false,"code_links":[{"ID":614170,"CreatedAt":"2026-07-14T01:21:01.169441415Z","UpdatedAt":"2026-07-14T01:21:01.169441415Z","DeletedAt":null,"paper_id":6536479,"paper_url":"https://arxiv.org/abs/2607.10391","paper_title":"Vertical Fusion: Condensing Internal Representations for Robust ViT Classification","repo_url":"https://github.com/francescodisalvo05/vit-vertical-fusion","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}