{"ID":6537529,"CreatedAt":"2026-07-14T02:54:43.516908796Z","UpdatedAt":"2026-07-15T03:28:55.185153975Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.11475","arxiv_id":"2607.11475","title":"HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models","abstract":"Safety alignment in large language models can be fragile under fine-tuning, as even benign task adaptation may increase harmful compliance. Existing defenses mainly follow two directions: they either intervene during or after fine-tuning through retraining or weight modification, which can be costly and may hurt task performance, or they use model-agnostic safety classifiers, which may miss failures specific to a given fine-tuned checkpoint. These limitations motivate a post hoc, model-specific, and non-invasive approach to safety restoration. To meet these requirements, we propose HyperSafe, a framework that restores safety behavior by generating a model-specific Safe Side Network (SSN) for each fine-tuned checkpoint. HyperSafe uses layer-wise activation fingerprints to capture how fine-tuning changes the model's inner representations. With a small set of given calibration prompts, the hypernetwork maps these fingerprints to the parameters of the \\ssn{} in a single forward pass. The generated \\ssn{} runs alongside the frozen fine-tuned model and performs prompt-level safety classification: harmful prompts are routed to refusal, while safe prompts are answered by the original fine-tuned model. Thus, HyperSafe requires no gradient updates, no safety data at deployment time, and no modification to the deployed model weights. We evaluate HyperSafe on two model families, Qwen2-7B and LLaMA-3-8B, across multiple safety benchmarks. HyperSafe reduces harmful response rates from 19-31% to below 1% on every held-out checkpoint, while keeping downstream task accuracy within 1% of the fine-tuned baseline on average. Code is available at https://github.com/nokronim/project-safety-remedy.","short_abstract":"Safety alignment in large language models can be fragile under fine-tuning, as even benign task adaptation may increase harmful compliance. Existing defenses mainly follow two directions: they either intervene during or after fine-tuning through retraining or weight modification, which can be costly and may hurt task p...","url_abs":"https://arxiv.org/abs/2607.11475","url_pdf":"https://arxiv.org/pdf/2607.11475v1","authors":"[\"Aznaur Aliev\",\"Carlos Hinojosa\",\"Abdelrahman Eldesokey\",\"Bang An\",\"Bernard Ghanem\",\"Yibo Yang\"]","published":"2026-07-13T12:28:43Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.CL\"]","methods":"[\"Language Model\"]","has_code":false,"code_links":[{"ID":614211,"CreatedAt":"2026-07-14T02:54:43.516908796Z","UpdatedAt":"2026-07-14T02:54:43.516908796Z","DeletedAt":null,"paper_id":6537529,"paper_url":"https://arxiv.org/abs/2607.11475","paper_title":"HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models","repo_url":"https://github.com/nokronim/project-safety-remedy","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}