{"ID":2898500,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2507.03779","arxiv_id":"2507.03779","title":"FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed","abstract":"Large-scale vision foundation models such as DINOv2 boast impressive performances by leveraging massive architectures and training datasets. But numerous scenarios require practitioners to reproduce those pre-training solutions, such as on private data, new modalities, or simply for scientific questioning--which is currently extremely demanding computation-wise. We thus propose a novel pre-training strategy for DINOv2 that simultaneously accelerates convergence--and strengthens robustness to common corruptions as a by-product. Our approach involves a frequency filtering curriculum--low-frequency being seen first--and the Gaussian noise patching augmentation. Applied to a ViT-B/16 backbone trained on ImageNet-1K, while pre-training time and FLOPs are reduced by 1.6x and 2.25x, our method still achieves matching robustness in corruption benchmarks (ImageNet-C) and maintains competitive linear probing performance compared with baseline. This dual benefit of efficiency and robustness makes large-scale self-supervised foundation modeling more attainable, while opening the door to novel exploration around data curriculum and augmentation as means to improve self-supervised learning models robustness. The code is available at https://github.com/KevinZ0217/fast_dinov2","short_abstract":"Large-scale vision foundation models such as DINOv2 boast impressive performances by leveraging massive architectures and training datasets. But numerous scenarios require practitioners to reproduce those pre-training solutions, such as on private data, new modalities, or simply for scientific questioning--which is cur...","url_abs":"https://arxiv.org/abs/2507.03779","url_pdf":"https://arxiv.org/pdf/2507.03779v3","authors":"[\"Jiaqi Zhang\",\"Juntuo Wang\",\"Zhixin Sun\",\"John Zou\",\"Randall Balestriero\"]","published":"2025-07-04T18:56:04Z","proceeding":"cs.CV","tasks":"[\"cs.CV\",\"cs.AI\",\"cs.LG\"]","methods":"[\"LoRA\"]","has_code":false,"code_links":[{"ID":612413,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2898500,"paper_url":"https://arxiv.org/abs/2507.03779","paper_title":"FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed","repo_url":"https://github.com/KevinZ0217/fast_dinov2","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}