{"ID":2897423,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2507.04776","arxiv_id":"2507.04776","title":"Improving BERT for Symbolic Music Understanding Using Token Denoising and Pianoroll Prediction","abstract":"We propose a pre-trained BERT-like model for symbolic music understanding that achieves competitive performance across a wide range of downstream tasks. To achieve this target, we design two novel pre-training objectives, namely token correction and pianoroll prediction. First, we sample a portion of note tokens and corrupt them with a limited amount of noise, and then train the model to denoise the corrupted tokens; second, we also train the model to predict bar-level and local pianoroll-derived representations from the corrupted note tokens. We argue that these objectives guide the model to better learn specific musical knowledge such as pitch intervals. For evaluation, we propose a benchmark that incorporates 12 downstream tasks ranging from chord estimation to symbolic genre classification. Results confirm the effectiveness of the proposed pre-training objectives on downstream tasks.","short_abstract":"We propose a pre-trained BERT-like model for symbolic music understanding that achieves competitive performance across a wide range of downstream tasks. To achieve this target, we design two novel pre-training objectives, namely token correction and pianoroll prediction. First, we sample a portion of note tokens and co...","url_abs":"https://arxiv.org/abs/2507.04776","url_pdf":"https://arxiv.org/pdf/2507.04776v1","authors":"[\"Jun-You Wang\",\"Li Su\"]","published":"2025-07-07T08:52:06Z","proceeding":"cs.SD","tasks":"[\"cs.SD\",\"cs.LG\",\"cs.MM\",\"eess.AS\"]","methods":"[]","has_code":false}