{"ID":2838074,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.20694","arxiv_id":"2511.20694","title":"Reasoning With a Star: A Heliophysics Dataset and Benchmark for Agentic Scientific Reasoning","abstract":"Scientific reasoning through Large Language Models in heliophysics involves more than just recalling facts: it requires incorporating physical assumptions, maintaining consistent units, and providing clear scientific formats through coordinated approaches. To address these challenges, we present Reasoning With a Star, a newly contributed heliophysics dataset applicable to reasoning; we also provide an initial benchmarking approach. Our data are constructed from National Aeronautics and Space Administration \u0026 University Corporation for Atmospheric Research Living With a Star summer school problem sets and compiled into a readily consumable question-and-answer structure with question contexts, reasoning steps, expected answer type, ground-truth targets, format hints, and metadata. A programmatic grader checks the predictions using unit-aware numerical tolerance, symbolic equivalence, and schema validation. We benchmark a single-shot baseline and four multi-agent patterns, finding that decomposing workflows through systems engineering principles outperforms direct prompting on problems requiring deductive reasoning rather than pure inductive recall.","short_abstract":"Scientific reasoning through Large Language Models in heliophysics involves more than just recalling facts: it requires incorporating physical assumptions, maintaining consistent units, and providing clear scientific formats through coordinated approaches. To address these challenges, we present Reasoning With a Star,...","url_abs":"https://arxiv.org/abs/2511.20694","url_pdf":"https://arxiv.org/pdf/2511.20694v2","authors":"[\"Kevin Lee\",\"Russell Spiewak\",\"James Walsh\"]","published":"2025-11-23T18:13:25Z","proceeding":"cs.AI","tasks":"[\"cs.AI\",\"astro-ph.SR\",\"cs.LG\",\"physics.space-ph\"]","methods":"[\"Language Model\"]","has_code":false}