In probability and statistics, the Dirichlet distribution (after Peter Gustav Lejeune Dirichlet), often denoted , is a family of continuous multivariate probability distributions parameterized by a vector of positive reals. It is a multivariate generalization of the beta distribution, (Chapter 49: Dirichlet and Inverted Dirichlet Distributions) hence its alternative name of multivariate beta distribution (MBD). Dirichlet distributions are commonly used as prior distributions in Bayesian statistics, and in fact, the Dirichlet distribution is the conjugate prior of the categorical distribution and multinomial distribution.
The infinite-dimensional generalization of the Dirichlet distribution is the Dirichlet process.
Definitions Probability density function
The Dirichlet distribution of order with parameters has a probability density function given by
where That is, the probability density function is defined on the standard simplex embedded in -dimensional Euclidean space, .
The normalizing constant is the multivariate beta function, which can be expressed in terms of the gamma function:
Support The support of the Dirichlet distribution is the set of -dimensional vectors whose entries are real numbers in the interval [0,1] such that , i.e. the sum of the coordinates is equal to 1. These can be viewed as the probabilities of a -way categorical event. Another way to express this is that the domain of the Dirichlet distribution is itself a set of probability distributions, specifically the set of -dimensional discrete distributions. The technical term for the set of points in the support of a -dimensional Dirichlet distribution is the open standard -simplex, which is a generalization of a triangle, embedded in the next-higher dimension. For example, with , the support is an equilateral triangle embedded in a downward-angle fashion in three-dimensional space, with vertices at (1,0,0), (0,1,0) and (0,0,1), i.e. touching each of the coordinate axes at a point 1 unit away from the origin.
Symmetric cases A common special case is the symmetric Dirichlet distribution, where all of the elements making up the parameter vector have the same value. The symmetric case might be useful, for example, when a Dirichlet prior over components is called for, but there is no prior knowledge favoring one component over another. Since all elements of the parameter vector have the same value, the symmetric Dirichlet distribution can be parametrized by a single scalar value , called the concentration parameter. In terms of , the density function has the form
When , the symmetric Dirichlet distribution is equivalent to a uniform distribution over the open standard -simplex, i.e. it is uniform over all points in its support. This particular distribution is known as the flat Dirichlet distribution. Values of the concentration parameter above 1 prefer variates that are dense, evenly distributed distributions, i.e. all the values within a single sample are similar to each other. Values of the concentration parameter below 1 prefer sparse distributions, i.e. most of the values within a single sample will be close to 0, and the vast majority of the mass will be concentrated in a few of the values.
When , the distribution is the same as would be obtained by choosing a point uniformly at random from the -dimensional unit hypersphere, which is the surface of a -dimensional unit hyperball, and squaring each coordinate. The distribution is the Jeffreys prior for the Dirichlet distribution.
With specified expectations Instead of specifying the vector of concentration parameters , one may wish to specify the expected probability distribution such that the concentration parameters are written as the product of the (scalar) concentration weight and the expected probability distribution whose components are non-negative and sum up to 1. The concentration weight in this case is larger by a factor of than the concentration parameter for a symmetric Dirichlet distribution described above. This construction ties in with concept of a base measure when discussing Dirichlet processes and is often used in the topic modelling literature.
When the dimension of the Dirichlet distribution is large, Jeffreys prior produces a correspondingly high prior concentration weight. For example when , the prior concentration weight becomes , which creates a relatively stiff prior probability distribution where initially the posterior probability distribution hardly changes in the presence of observed data, even in the case of e.g. 10 observations where all observations indicate the same category , which naturally and intuitively should produce an expected probability close to 1. A way to avoid the misbehaviour of the stiff prior is to use the noninformative prior weight from subjective logic, where the concentration parameters are expressed as a function of the observations according to .A. Jøsang, J.H. Cho, and F. Chen. Noninformative Prior Weights for Dirichlet PDFs. Proceedings of the 2022 IEEE International Conference on Multisensor Fusion and Integration (MFI 2022), Cranfield, UK, September 2022. PDF The noninformative prior concentration weight is a function of the number of observations , expressed as
where is a convergence constant, normally chosen to be equal to 2. The choice of ensures that the Dirichlet distribution of any dimension has the same sensitivity to new observations as the Beta distribution with prior concentration parameters (or the uniform ). In the case of a uniform prior probability distribution , the noninformative prior weight also ensures a uniform prior Dirichlet distribution for any dimension . A 3D plot of is illustrated in the figure.
Properties
Moments Let .
Let
ThenEq. (49.9) on page 488 of Kotz, Balakrishnan & Johnson (2000). Continuous Multivariate Distributions. Volume 1: Models and Applications. New York: Wiley.
Furthermore, if
The covariance matrix is singular.
More generally, moments of Dirichlet-distributed random variables can be expressed in the following way. For , denote by its -th Hadamard power. Then,
where the sum is over non-negative integers with , and is the cycle index polynomial of the Symmetric group of degree .
We have the special case
The multivariate analogue for vectors can be expressed in terms of a color pattern of the exponents in the sense of the Pólya enumeration theorem.
Particular cases include the simple computation
Mode The mode of the distribution is the vector with
Marginal distributions The marginal distributions are beta distributions:
Also see below.
Conjugate to categorical or multinomial The Dirichlet distribution is the conjugate prior distribution of the categorical distribution (a generic discrete probability distribution with a given number of possible outcomes) and multinomial distribution (the distribution over observed counts of each possible category in a set of categorically distributed observations). This means that if a data point has either a categorical or multinomial distribution, and the prior distribution of the distribution's parameter (the vector of probabilities that generates the data point) is distributed as a Dirichlet, then the posterior distribution of the parameter is also a Dirichlet. Intuitively, in such a case, starting from what we know about the parameter prior to observing the data point, we then can update our knowledge based on the data point and end up with a new distribution of the same form as the old one. This means that we can successively update our knowledge of a parameter by incorporating new observations one at a time, without running into mathematical difficulties.
Formally, this can be expressed as follows. Given a model
then the following holds:
This relationship is used in Bayesian statistics to estimate the underlying parameter of a categorical distribution given a collection of samples. Intuitively, we can view the hyperprior vector as pseudocounts, i.e. as representing the number of observations in each category that we have already seen. Then we simply add in the counts for all the new observations (the vector ) in order to derive the posterior distribution.
In Bayesian mixture models and other hierarchical Bayesian models with mixture components, Dirichlet distributions are commonly used as the prior distributions for the categorical variables appearing in the models. See the section on applications below for more information.
Relation to Dirichlet-multinomial distribution In a model where a Dirichlet prior distribution is placed over a set of categorical-valued observations, the marginal joint distribution of the observations (i.e. the joint distribution of the observations, with the prior parameter marginalized out) is a Dirichlet-multinomial distribution. This distribution plays an important role in hierarchical Bayesian models, because when doing inference over such models using methods such as Gibbs sampling or variational Bayes, Dirichlet prior distributions are often marginalized out. See the article on this distribution for more details.
Entropy
If is a random variable, the differential entropy of (in nat units) is
where is the digamma function.
The following formula for can be used to derive the differential entropy above. Since the functions are the sufficient statistics of the Dirichlet distribution, the exponential family differential identities can be used to get an analytic expression for the expectation of (see equation (2.62) in ) and its associated covariance matrix:
and
where is the digamma function, is the trigamma function, and is the Kronecker delta.
The spectrum of Rényi information for values other than is given by
and the information entropy is the limit as goes to 1.
Another related interesting measure is the entropy of a discrete categorical (one-of-K binary) vector with probability-mass distribution , i.e., . The conditional information entropy of , given is
This function of is a scalar random variable. If has a symmetric Dirichlet distribution with all , the expected value of the entropy (in nat units) is, eq. 8
Kullback–Leibler divergence
The Kullback–Leibler (KL) divergence between two Dirichlet distributions, and , over the same simplex is:
Aggregation If
then, if the random variables with subscripts and are dropped from the vector and replaced by their sum,
This aggregation property may be used to derive the marginal distribution of mentioned above.
Neutrality
If , then the vector is said to be neutral in the sense that X is independent of where
and similarly for removing any of . Observe that any permutation of is also neutral (a property not possessed by samples drawn from a generalized Dirichlet distribution).See Kotz, Balakrishnan & Johnson (2000), Section 8.5, "Connor and Mosimann's Generalization", pp. 519–521.
Combining this with the property of aggregation it follows that is independent of . In fact it is true, further, for the Dirichlet distribution, that for , the pair , and the two vectors and , viewed as triple of normalised random vectors, are mutually independent. The analogous result is true for partition of the indices into any other pair of non-singleton subsets.
Characteristic function
The characteristic function of the Dirichlet distribution is a confluent form of the Lauricella hypergeometric series. It is given by Phillips as
where
The sum is over non-negative integers and . Phillips goes on to state that this form is "inconvenient for numerical calculation" and gives an alternative in terms of a complex path integral:
where denotes any path in the complex plane originating at , encircling in the positive direction all the singularities of the integrand and returning to .
Inequality Probability density function plays a key role in a multifunctional inequality which implies various bounds for the Dirichlet distribution.