Avoiding Non-Integrable Beliefs in Expectation Propagation

arXiv stat.ML / 4/7/2026

💬 OpinionIdeas & Deep AnalysisModels & Research

共有:

Key Points

Expectation Propagation (EP) can be viewed as an iterative optimizer of a constrained Bethe Free Energy (BFE), but the iteration can move outside the feasible set, producing non-integrable “beliefs.”
Prior work typically enforces integrability by constraining messages, yet this may unduly shrink the feasible set in many Bayesian estimation settings.
The paper argues that even making messages integrable may fail to guarantee integrable beliefs when factors themselves are non-integrable.
It introduces two EP frameworks designed to ensure integrable beliefs while still allowing non-integrable messages.
The proposed methods are evaluated on a signal recovery task in Generalized Linear Models (GLMs).

Abstract

Expectation Propagation (EP) is a widely used iterative message-passing algorithm that decomposes a global inference problem into multiple local ones. It approximates marginal distributions as ``beliefs'' using intermediate functions called ``messages''. It has been shown that the stationary points of EP are the same as corresponding constrained Bethe Free Energy (BFE) optimization problem. Therefore, EP is an iterative method of optimizing the constrained BFE. However, the iterative method may fall out of the feasible set of the BFE optimization problem, i.e., the beliefs are not integrable. In most literature, the authors use various methods to keep all the messages integrable. In most Bayesian estimation problems, limiting the messages to be integrable shrinks the actual feasible set. Furthermore, in extreme cases where the factors are not integrable, making the message itself integrable is not enough to have integrable beliefs. In this paper, two EP frameworks are proposed to ensure that EP has integrable beliefs. Both of the methods allows non-integrable messages. We then investigate the signal recovery problem in Generalized Linear Model (GLM) using our proposed methods.