{"ID":2872600,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.09732","arxiv_id":"2509.09732","title":"Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs","abstract":"Vision language models (VLMs) excel at zero-shot visual classification, but their performance on fine-grained tasks and large hierarchical label spaces is understudied. This paper investigates whether structured, tree-based reasoning can enhance VLM performance. We introduce a framework that decomposes classification into interpretable decisions using decision trees and evaluates it on fine-grained (GTSRB) and coarse-grained (CIFAR-10) datasets. Although the model achieves 98.2% accuracy in understanding the tree knowledge, tree-based reasoning consistently underperforms standard zero-shot prompting. We also explore enhancing the tree prompts with LLM-generated classes and image descriptions to improve alignment. The added description enhances the performance of the tree-based and zero-shot methods. Our findings highlight limitations of structured reasoning in visual classification and offer insights for designing more interpretable VLM systems.","short_abstract":"Vision language models (VLMs) excel at zero-shot visual classification, but their performance on fine-grained tasks and large hierarchical label spaces is understudied. This paper investigates whether structured, tree-based reasoning can enhance VLM performance. We introduce a framework that decomposes classification i...","url_abs":"https://arxiv.org/abs/2509.09732","url_pdf":"https://arxiv.org/pdf/2509.09732v1","authors":"[\"Sary Elmansoury\",\"Islam Mesabah\",\"Gerrit Großmann\",\"Peter Neigel\",\"Raj Bhalwankar\",\"Daniel Kondermann\",\"Sebastian J. Vollmer\"]","published":"2025-09-10T13:08:03Z","proceeding":"cs.CV","tasks":"[\"cs.CV\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}