{"ID":2922042,"CreatedAt":"2026-06-02T02:42:49.606572591Z","UpdatedAt":"2026-06-02T09:31:37.03791814Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.00644","arxiv_id":"2606.00644","title":"ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment","abstract":"AI research often requires decisions before future evidence exists: which bottleneck to attack, which direction to pursue, or where a project should be positioned. We introduce ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical evidence. ForeSci contains 500 tasks across four fast-moving AI domains and four decision families. Each task is paired with a cutoff-aligned offline knowledge base; post-cutoff papers are hidden during generation and used only for validation. To avoid random future-event prediction, tasks are derived from pre-cutoff taxonomy branches and evidence signals, and answer-generation backbones are selected to precede the task cutoffs. We evaluate native LLMs, Hybrid RAG, and three research-agent adaptations across four backbones. Results show that explicit evidence organization improves traceability and factual support, but gains depend strongly on the decision family. Diagnostics reveal a recurring evidence-decision decoupling: agents may cite relevant evidence while forecasting the wrong research object. ForeSci turns forward-looking AI research judgement into a controlled benchmark for evaluating research agents as decision-making systems.","short_abstract":"AI research often requires decisions before future evidence exists: which bottleneck to attack, which direction to pursue, or where a project should be positioned. We introduce ForeSci, a temporally controlled benchmark for evaluating whether LLM agents can make such forward-looking research judgements from historical...","url_abs":"https://arxiv.org/abs/2606.00644","url_pdf":"https://arxiv.org/pdf/2606.00644v1","authors":"[\"Qiuyu Tian\",\"Zequn Liu\",\"Yingce Xia\",\"Haojie Yin\",\"Youyong Kong\"]","published":"2026-05-30T09:41:26Z","proceeding":"cs.AI","tasks":"[\"cs.AI\"]","methods":"[\"Large Language Model\",\"Generative Adversarial Network\"]","has_code":false}