{"ID":104043,"CreatedAt":"2026-02-27T13:00:40Z","UpdatedAt":"2026-02-27T13:00:40Z","DeletedAt":null,"paper_url":"https://paperswithcode.com/paper/the-information-bottleneck-method","arxiv_id":"physics/0004057","title":"The information bottleneck method","abstract":"We define the relevant information in a signal $x\\in X$ as being the information that this signal provides about another signal $y\\in \\Y$. Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. Understanding the signal $x$ requires more than just predicting $y$, it also requires specifying which features of $\\X$ play a role in the prediction. We formalize this problem as that of finding a short code for $\\X$ that preserves the maximum information about $\\Y$. That is, we squeeze the information that $\\X$ provides about $\\Y$ through a `bottleneck' formed by a limited set of codewords $\\tX$. This constrained optimization problem can be seen as a generalization of rate distortion theory in which the distortion measure $d(x,\\x)$ emerges from the joint statistics of $\\X$ and $\\Y$. This approach yields an exact set of self consistent equations for the coding rules $X \\to \\tX$ and $\\tX \\to \\Y$. Solutions to these equations can be found by a convergent re-estimation method that generalizes the Blahut-Arimoto algorithm. Our variational principle provides a surprisingly rich framework for discussing a variety of problems in signal processing and learning, as will be described in detail elsewhere.","short_abstract":"We define the relevant information in a signal $x\\in X$ as being the information that this signal provides about another signal $y\\in \\Y$.","url_abs":"https://arxiv.org/abs/physics/0004057v1","url_pdf":"https://arxiv.org/pdf/physics/0004057v1.pdf","authors":"[\"Naftali Tishby\", \"Fernando C. Pereira\", \"William Bialek\"]","published":"2000-04-24T00:00:00Z","tasks":"[]","methods":"[]","has_code":false,"code_links":[{"ID":72610,"CreatedAt":"2026-02-27T13:01:31Z","UpdatedAt":"2026-02-27T13:01:31Z","DeletedAt":null,"paper_id":104043,"paper_url":"https://paperswithcode.com/paper/the-information-bottleneck-method","paper_title":"The information bottleneck method","repo_url":"https://github.com/sepehr-rasouli/ITRL","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"framework":"none","github_stars":0},{"ID":85178,"CreatedAt":"2026-02-27T13:01:31Z","UpdatedAt":"2026-02-27T13:01:31Z","DeletedAt":null,"paper_id":104043,"paper_url":"https://paperswithcode.com/paper/the-information-bottleneck-method","paper_title":"The information bottleneck method","repo_url":"https://github.com/sepehr-rasouli/PCBS-ITRL","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"framework":"none","github_stars":0},{"ID":197687,"CreatedAt":"2026-02-27T13:01:31Z","UpdatedAt":"2026-02-27T13:01:31Z","DeletedAt":null,"paper_id":104043,"paper_url":"https://paperswithcode.com/paper/the-information-bottleneck-method","paper_title":"The information bottleneck method","repo_url":"https://github.com/vursekar/information-bottleneck-on-a-grid","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"framework":"none","github_stars":0},{"ID":281219,"CreatedAt":"2026-02-27T13:01:31Z","UpdatedAt":"2026-02-27T13:01:31Z","DeletedAt":null,"paper_id":104043,"paper_url":"https://paperswithcode.com/paper/the-information-bottleneck-method","paper_title":"The information bottleneck method","repo_url":"https://github.com/Yuhan556/RDPC","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"framework":"pytorch","github_stars":0},{"ID":372771,"CreatedAt":"2026-03-04T21:00:12Z","UpdatedAt":"2026-03-04T21:00:12Z","DeletedAt":null,"paper_id":104043,"paper_url":"https://paperswithcode.com/paper/the-information-bottleneck-method","paper_title":"The information bottleneck method","repo_url":"https://github.com/sepehr-rasouli/ITRL","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"framework":"none","github_stars":0},{"ID":385339,"CreatedAt":"2026-03-04T21:00:12Z","UpdatedAt":"2026-03-04T21:00:12Z","DeletedAt":null,"paper_id":104043,"paper_url":"https://paperswithcode.com/paper/the-information-bottleneck-method","paper_title":"The information bottleneck method","repo_url":"https://github.com/sepehr-rasouli/PCBS-ITRL","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"framework":"none","github_stars":0},{"ID":497848,"CreatedAt":"2026-03-04T21:00:12Z","UpdatedAt":"2026-03-04T21:00:12Z","DeletedAt":null,"paper_id":104043,"paper_url":"https://paperswithcode.com/paper/the-information-bottleneck-method","paper_title":"The information bottleneck method","repo_url":"https://github.com/vursekar/information-bottleneck-on-a-grid","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"framework":"none","github_stars":0},{"ID":581380,"CreatedAt":"2026-03-04T21:00:12Z","UpdatedAt":"2026-03-04T21:00:12Z","DeletedAt":null,"paper_id":104043,"paper_url":"https://paperswithcode.com/paper/the-information-bottleneck-method","paper_title":"The information bottleneck method","repo_url":"https://github.com/Yuhan556/RDPC","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"framework":"pytorch","github_stars":0}]}