{"ID":6536244,"CreatedAt":"2026-07-14T01:21:01.169441415Z","UpdatedAt":"2026-07-15T03:28:55.185153975Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.10806","arxiv_id":"2607.10806","title":"Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation","abstract":"Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond surface-level metrics like ROUGE. We introduce Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR) -- a set of principled heuristic metrics that measure how much a summary diverges from extractive copying of the source text. The formulation uses the harmonic mean of document lengths modulated by a cubic non-overlap factor, yielding dimensionally consistent, bounded output with non-linear sensitivity to the extractive-abstractive boundary. Evaluation on 100 XSUM documents across four summarization models (BART-large-cnn, Pegasus-xsum, DistilBart, MT5-small) demonstrates that the metrics successfully discriminate between extractive models (SA ~ 0.12-0.26) and abstractive models (SA ~ 0.96-1.77), and that the Abstraction Ratio identifies summaries requiring manual evaluation for potential hallucination. Code and results are available at https://github.com/katweNLP/AbstractionStudy.","short_abstract":"Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond surface-level metrics like ROUGE. We introduce Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR) -- a set of principled heuristic metrics that measure how much a summary diverges fro...","url_abs":"https://arxiv.org/abs/2607.10806","url_pdf":"https://arxiv.org/pdf/2607.10806v1","authors":"[\"Praveenkumar Katwe\",\"Rakesh Chandra Balabantaray\",\"Kali Prasad Vittala\"]","published":"2026-07-12T15:25:57Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\"]","methods":"[\"Convolutional Neural Network\"]","has_code":false,"code_links":[{"ID":614151,"CreatedAt":"2026-07-14T01:21:01.169441415Z","UpdatedAt":"2026-07-14T01:21:01.169441415Z","DeletedAt":null,"paper_id":6536244,"paper_url":"https://arxiv.org/abs/2607.10806","paper_title":"Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation","repo_url":"https://github.com/katweNLP/AbstractionStudy","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}