Exponential Family - The Meaning of "exponential Family"

The Meaning of "exponential Family"

It is critical, when considering the above definitions, to use proper terminology and to keep in mind exactly what is being spoken of when the term "exponential family" is used. Properly speaking, there is no such thing as "the" exponential family, but rather an exponential family, and properly speaking, it is not a "distribution" but a family of distributions that either is or is not an exponential family. The problem lies in the fact that we often say, e.g., "the normal distribution" when properly we mean something like "the family of normal distributions with unknown mean and variance". A family of distributions is defined by a set of parameters that can be varied, and what makes a family be an exponential family is a particular relationship between the domain of a family of distributions (the variable over which each distribution in the family is defined) and the parameters.

As an example, what is often said to be "the binomial distribution" is in fact a family of related distributions characterized by a parameter n of Bernoulli trials, each of which is drawn using a parameter p (a probability of success). A particular setting of n and p characterizes a particular probability distribution over a discrete random variable, with possible outcomes (the support of the distribution) ranging between 0 and n. Consider the following cases:

1. If both n and p are given particular settings (e.g. n=20, p=0.1), a single binomial distribution arises.
2. If n is given a particular setting (e.g. n=20), but p is allowed to vary, a family of binomial distributions arises, characterized by the parameter p.
3. If both n and p are allowed to vary, a different (and larger) family of binomial distributions arises, characterized by the parameters n and p.

All of the above cases can be referred to using the term "binomial distribution", but not all of them are exponential families. In fact, only the second one is an exponential family:

• The first case (with fixed n and p) is not a family of distributions at all, but a single distribution, and hence cannot logically be an exponential family.
• The third case happens not to be an exponential family. In general, exponential families cannot have a support that varies according to a parameter; rather, the support must remain the same across all distributions in the family.

Even more confusing is the case of the uniform distribution. It is common to say something like "draw a number from a uniform distribution" to mean specifically to draw a number from a continuous uniform distribution that ranges between 0 and 1. Similarly, it is sometimes said that "the uniform distribution is a special case of the beta distribution", again referring to a continuous uniform distribution ranging between 0 and 1. Since the beta distribution is an exponential family, it is tempting to conclude that the uniform distribution is also an exponential family. In fact, however, both examples above refer to a specific uniform distribution, not a family. The family of uniform distributions is defined by either an unknown upper bound, unknown lower bound, or unknown upper and lower bounds — and none of these families are exponential families. (This can be seen by considering what was said above — the support of an exponential family cannot vary depending on a particular parameter.) Hence, it is often said that the "uniform distribution" is not an exponential family, which is correct but imprecise.