{"ID":5443779,"CreatedAt":"2026-07-01T02:07:11.383974684Z","UpdatedAt":"2026-07-03T14:25:27.813080916Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.31734","arxiv_id":"2606.31734","title":"MemLearner: Learning to Query Context memory for Video World Models","abstract":"Video World Models are interactive video generation models that predict future world states based on user actions and history video frames. A critical challenge in video world models is the lack of memory, causing inconsistent generated scenes over extended durations. Previous methods explored rule-based context frame retrieval as memory, but they fail to generalize in scenarios with scene occlusions and dynamic objects. We propose MemLearner, a learning-based adaptive context query method using query tokens to bridge context and predicted tokens. By leveraging the video generation model itself for context querying, MemLearner exploits pre-trained visual priors without training additional modules from scratch, and incorporates efficient strategies for training and inference. We collect a dataset of long videos with scene occlusions and dynamic objects, paired with camera pose annotations, and propose a multi-dataset training strategy leveraging both annotated rendered and unannotated real-world videos. Extensive experiments demonstrate that MemLearner significantly outperforms prior video world models in terms of scene consistency and memory, particularly under challenging occlusion and dynamic scenarios.","short_abstract":"Video World Models are interactive video generation models that predict future world states based on user actions and history video frames. A critical challenge in video world models is the lack of memory, causing inconsistent generated scenes over extended durations. Previous methods explored rule-based context frame...","url_abs":"https://arxiv.org/abs/2606.31734","url_pdf":"https://arxiv.org/pdf/2606.31734v1","authors":"[\"Jiwen Yu\",\"Jianxiong Gao\",\"Jianhong Bai\",\"Yiran Qin\",\"Kaiyi Huang\",\"Quande Liu\",\"Xintao Wang\",\"Pengfei Wan\",\"Kun Gai\",\"Xihui Liu\"]","published":"2026-06-30T14:31:32Z","proceeding":"cs.CV","tasks":"[\"cs.CV\"]","methods":"[]","has_code":false}