{"ID":3053356,"CreatedAt":"2026-06-04T04:41:36.695875263Z","UpdatedAt":"2026-06-06T03:14:50.67780443Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.04394","arxiv_id":"2606.04394","title":"Beyond Single-Policy: Evaluating Composed Organization-Specific Policy Alignment in LLM Chatbots","abstract":"Large language model chatbots are increasingly deployed in organizational settings such as healthcare, finance, and public services. Evaluating policy alignment is therefore critical to reliable chatbot deployment. By analyzing real-world user queries, we identify composed-policy violation is prevalent in various chatbots but overlooked by existing benchmarks. This paper present COPAL, an automated tool for evaluating composed-policy alignment in chatbots. COPAL efficiently generates queries that trigger composed-policy failures in chatbots via empirically derived interaction patterns and explicit handling contracts. Queries generated by COPAL expose substantial query handling failures: across 9 served models, composed-policy queries yield a 33.1% error rate on average, indicating that composed-policy alignment warrants further investigation.","short_abstract":"Large language model chatbots are increasingly deployed in organizational settings such as healthcare, finance, and public services. Evaluating policy alignment is therefore critical to reliable chatbot deployment. By analyzing real-world user queries, we identify composed-policy violation is prevalent in various chatb...","url_abs":"https://arxiv.org/abs/2606.04394","url_pdf":"https://arxiv.org/pdf/2606.04394v1","authors":"[\"Yingjie Liu\",\"Yongxiang Hu\",\"Xuan Wang\",\"Yilun Li\",\"Yunlei Wei\",\"Xiaoyu Wang\",\"Yangfan Zhou\"]","published":"2026-06-03T03:17:26Z","proceeding":"cs.SE","tasks":"[\"cs.SE\"]","methods":"[\"Large Language Model\",\"Language Model\",\"Generative Adversarial Network\"]","has_code":false}