{"ID":5937935,"CreatedAt":"2026-07-07T03:14:33.014478982Z","UpdatedAt":"2026-07-07T09:57:39.419301368Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.03801","arxiv_id":"2607.03801","title":"Punching Above Their Weight: Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks","abstract":"We define Tiny Language Models (TLMs) as models below roughly 3B parameters that fit on mainstream consumer devices. We study how to adapt them for and use them on verifiable multiple-choice tasks. We compare three LoRA-based fine-tuning paradigms (label generation, gold only, and our discriminative classification head) on a unified setup across several Qwen3 models from 0.6B to 8B and five benchmarks: HellaSwag, WinoGrande, PIQA, SciQ and ARC-C. Classification-head fine-tuning reliably outperforms label generation (+2-3%) at the 0.6B and 1.7B scales. Further, TLMs fine-tuned using the discriminative method are competitive to zero-/few-shot GPT-3 (175B), PaLM (540B) and GPT-4. The performance we report for Qwen3-0.6B and Qwen3-1.7B are SOTA on HellaSwag, WinoGrande, and PIQA.","short_abstract":"We define Tiny Language Models (TLMs) as models below roughly 3B parameters that fit on mainstream consumer devices. We study how to adapt them for and use them on verifiable multiple-choice tasks. We compare three LoRA-based fine-tuning paradigms (label generation, gold only, and our discriminative classification head...","url_abs":"https://arxiv.org/abs/2607.03801","url_pdf":"https://arxiv.org/pdf/2607.03801v1","authors":"[\"Bhavesh Sood\",\"Jaromir Savelka\"]","published":"2026-07-04T10:09:32Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\",\"cs.CL\"]","methods":"[\"Language Model\",\"LoRA\"]","has_code":false}