{"ID":2898964,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2507.01271","arxiv_id":"2507.01271","title":"PULSE: Practical Evaluation Scenarios for Large Multimodal Model Unlearning","abstract":"In recent years, unlearning techniques, which are methods for inducing a model to \"forget\" previously learned information, have attracted attention as a way to address privacy and copyright concerns in large language models (LLMs) and large multimodal models (LMMs). While several unlearning benchmarks have been established for LLMs, a practical evaluation framework for unlearning in LMMs has been less explored. Specifically, existing unlearning benchmark for LMMs considers only scenarios in which the model is required to unlearn fine-tuned knowledge through a single unlearning operation. In this study, we introduce PULSE protocol for realistic unlearning scenarios for LMMs by introducing two critical perspectives: (i) Pre-trained knowledge Unlearning for analyzing the effect across different knowledge acquisition phases and (ii) Long-term Sustainability Evaluation to address sequential requests. We then evaluate existing unlearning methods along these dimensions. Our results reveal that, although some techniques can successfully unlearn knowledge acquired through fine-tuning, they struggle to eliminate information learned during pre-training. Moreover, methods that effectively unlearn a batch of target data in a single operation exhibit substantial performance degradation when the same data are split and unlearned sequentially.","short_abstract":"In recent years, unlearning techniques, which are methods for inducing a model to \"forget\" previously learned information, have attracted attention as a way to address privacy and copyright concerns in large language models (LLMs) and large multimodal models (LMMs). While several unlearning benchmarks have been establi...","url_abs":"https://arxiv.org/abs/2507.01271","url_pdf":"https://arxiv.org/pdf/2507.01271v4","authors":"[\"Tatsuki Kawakami\",\"Kazuki Egashira\",\"Atsuyuki Miyai\",\"Go Irie\",\"Kiyoharu Aizawa\"]","published":"2025-07-02T01:13:08Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}