7. The Gauss Probability Distribution

Introduction

We have already discussed one approximation (or limit) of the Binomial probability distribution in module 5 namely, the Poisson distribution.  The Poisson limit assumed one of the experimental probabilities p is quite small (and the other q is almost certain).  This is a situation that occurs in an experiment like the radioactive decay of a sample of atoms.  During a particular time interval the probability of decay of an atom p is small and so p<<1.  Alternatively the probability q that an atom does not decay is almost certain so q≈1.
	Here instead we discuss another important limit of the Binomial distribution valid.  This Gauss approximation to the Binomial probability distribution is valid in a wide variety of experimental situations where the random error are equally likely in the positive as well as negative direction about the average experimental reading.  Experimentalists like to use the Gauss distribution instead of the Binomial distribution because the Gauss distribution is easier to handle computationally.  The factorial functions in the Binomail distribution require a good deal of computational effort for N>100,000 say.  The Gauss distribution involves exponential functions which are a good deal easier to compute than factorial functions.

The Binomial Probability Distribution

Recall (from module 4) the Binomial Probability Distribution W[(Subscript[n, R])] of a drunk taking (Subscript[n, R]) steps to the right (and (Subscript[n, L]) steps to the left)

\(\(\(\ \ \)\(W[
        n\_R] = \ \(N!\)\/\(\(n\_R!\)\ \(n\_L!\)\)\ \ p\^n\_R\ \ q\^n\_L\)\)\)

N is the total number of steps in the drunk takes and p is the probability the drunk takes a step to the right (and q is the probability the drunk takes as step to the left).  We know that the total number of steps N=(Subscript[n, R])+(Subscript[n, L]) and also that p+q=1 because there are only two possibilities, either the drunk takes a step to the right or to the left.
	We are not so much interested in the (random) walk of drunks as we are in laboratory situations where the probability of the random errors must be included in the report of a measurement.  Many laboratory situations involve cases where one of the probabilities, say p approximately 0.5  (p≈0.5) and as a consequence q≈0.5 (since q=1-p).  This limit of the Binomial probability distribution (for N large) is called the Gauss probability distribution and this is the subject of this module.
	We simplify the notation in equation (1) slightly by using n instead of (Subscript[n, R]) so that (Subscript[n, L])=N-(Subscript[n, R])=N-n  instead.  The Binomial distribution thus appears in the simpler form

\(\(\(\ \ \ \)\(W[
        n] = \ \(N!\)\/\(\(n!\) \(\((\ N - n)\)!\)\)\ \ p\^n\ \ q\^\(N - n\)\)\)\)

Observation about the case p≈0.5

A typical case of the Binomial distribution for p=0.5  is

Num = 100;
p = 0.5;
q = 1 - p; 
W[n_] :=  ((Num!)/(n! (Num - n)!))  (p^n)   (q^(Num - n))

data = Table[{n, W[n]}, {n, 0, Num}];
ListPlot[data, PlotRange -> {0, 0.1}, 
      PlotStyle -> PointSize[0.01], 
      AxesLabel -> {"n", "W[n]"}]

The range of the probability distribution is 0≤n≤N but notice that for most values of n , W[n] is zero.  The only place where W[n]is not very small is when n≈ (N/2) (say  40≤n≤60  in the above example).

It is also important to observe from the graph, that the function W[n] while not continuous is more enough smooth when N is very large.  In particular this means that W[n] change little with n that is, W[m+1]-W[n]<<W[n] and in practice we assume W[n] is smooth enough to compute a derivative of W[n].  We will use the above observations to obtain the Gauss approximation to the Binomial distribution.

Some Approximations for p≈0.5 and N large

The numerical example above indicates the only place where W[n] is non-zero is in a small region about n≈(N/2) so it makes sense in approximations to focus on this region. Since Taylor series are good at approximating function the first idea that comes to mind is to expand W[n] about the peak value of n which we call \(TraditionalForm\`n\&_\)

\(\(\(\ \)\(W[n] = 
      W[n\&_]\  + \ \(d\/dn\) 
            W[n]\( | \_\(n = n\&_\)\)\ \((n - n\&_)\) + \ \(d\^2\/dn\^2\) 
            W[n]\( | \_\(n = 
              n\&_\)\)\ \((n - n\&_)\)\^2\/\(2!\) + \ \(\(·\)\(·\)\(·\)\)\)\)\)

(Recall the appendix on Taylor series in module 5 on the Poisson distribution.  Here the function expanded is f[n]=W[n] and the x variable of module 4 is replaced by the variable n here.)

The problem with expansion (3) is that W[n] of the Binomial distribution involves the calculation of factorial functions n! and (N-n)! which are in general quite large.  What is more important for the moment, is that the values of these factorial functions change by huge amounts in the region where W[n] is not zero.

For the numerical example above, at the peak where n= (100/2)=50 we compute

Num = 100. ;
n = (Num/2);
n!
(Num - n)!

and both have the same size.  However, on the LHS of the peak, n≈40 so the same factorials are

n = 52;
SetPrecision[n!, 6]
(Num - n)!

Thus there is a huge difference since n! has increased by three orders of magnitude and (N-n)! has decreased by three orders of magnitude.  Since W[n] involves factorial functions, the calculation of the derivatives of W[n] in the Taylor series (3) presents technical (or mathematical) difficulties.

These considerations cause us to consider a different approximation, and instead we consider a Taylor series  of the function LogW[n].  First note that the peak of W[n] is at the same place \(TraditionalForm\`n\&_\)  as the peak of LogW[n] since

(d/dn) LogW[n] = (1/W)[n] (d/dn) 
        W[n]

Since W[n] is obviously not zero at the peak \(TraditionalForm\`n\&_\), it is clear if (d/dn) W[n] = 0 then (d/dn) LogW[n]=0 and vice versa.

Instead of the Taylor expansion (3) we consider the Taylor expansion of f[n]=LogW[n]

\(\(\(\ \)\(Log\ W[n] = 
      Log\ W[n\&_]\  + \ \(d\/dn\) 
            Log\ W[n]\( | \_\(n = n\&_\)\)\ \((n - 
              n\&_)\) + \ 
\(d\^2\/dn\^2\) 
            Log\ W[n]\( | \_\(n = 
              n\&_\)\)\ \((n - n\&_)\)\^2\/\(2!\) + \ \(\(·\)\(·\)\(·\)\)\)\)\)

Taking the Log of equation (2) and using the propery of the Log function Log[
          x y] =  Log[x]+Log[y] , it is easily so see that the LogW[n] may be also written in the form

LogW[n] = 
      Log N! - Log n!  -  Log (N - n)!  +  
        n Log p  +  (N - n) Log q

so calculating derivatives of LogW[n] involves taking derivatives of the Log of the factorial function.  For example, using the numerical approximation of the derivative

\(\(d\/dn\) 
          Log\ \(n!\) ≈ \(Log\ \(\((n + 1)\)!\) - Log\ \(n!\)\)\/1 = \(Log\ \(\((n + 1)\)!\)\/\(n!\) = \ 
        Log\ \((n + 1)\) ≈ Log\ n\)\)

where the second equality involved the use of a property of the Log function, namely Log[(x/y)] =  Log[x] - Log[y].

We are now in a position to calculate the first derivative in the Taylor expansion (5).  In particular, after noting N is constant

(d/dn) 
        LogW[n] =  -(d/dn) Log n! -  (d/dn) 
          Log (N - n)! + Log p  -  
        Log q

Using equation (7) for the derivative of the factorial function we obtain approximately

(d/dn) LogW[n] ≈  -Log n +  
        Log (N - n) + Log p  -  Log q

and again using a property of the Log function

(d/dn) LogW[n] ≈  
      Log [(((N - n) p)/nq)]

The value of the first derivative is zero at the maximum \(TraditionalForm\`n\&_\) so

\(\(\(\ \)\(Log\ [\(\((N - n\&_)\) p\)\/\(n\&_\ q\)] = \(0\ \ \ \ \ or\ \ \ \ \ \ [\(\((N - n\&_)\) p\)\/\(n\&_\ q\)] = 1\)\)\)\)

since the only place where the Log function is zero is when the argument of the Log function is one.  Equation (11) allows us to compute n at the maximum, namely \(TraditionalForm\`n\&_\) since now

\(\(\(\ \)\(\((N - n\&_)\) p = n\&_\ q\)\)\)

and after rearrangement one obtains

\(\(\(\ \)\(n\&_\  = \ N\ p\)\)\)

where p+q=1 was utilized.  The Taylor expansion (5) of LogW[n] also requires the second derivative which from (9) is

((d^2)/(dn^2)) LogW[n] ≈  - (1/n) -  
        (1/((N - n)))

The second derivative is evaluated at \(TraditionalForm\`n\&_\) in the Taylor expansion (5) so

\(\(\(\ \)\(\(d\^2\/dn\^2\) 
          LogW[n]\( | \_\(n = 
              n\&_\)\)\(\(≈\)\(\ \)\(\(-\ \(1\/n\&_\)\) - \ 
            1\/\((N - n\&_)\)\)\) = \(-\ \(1\/\(N\ p\)\)\) - \ 
        1\/\((N - N\ p)\)\)\)\)

and after a little algebra

\(\(\(\ \ \)\(\(\(\(d\^2\/dn\^2\) 
          LogW[n]\)\( | \_\(n = n\&_\)\)\) = \(\(-\ \(1\/\(N\ p\)\)\) - \ 
          1\/\(N \((1 - \ p)\)\) = \ \(-\ \(1\/\(N\ p\)\)\) - \ 
          1\/\(N\ q\)\)\)\)\)

and further since q+p=1

\(\(\(\ \)\(\(\(\(d\^2\/dn\^2\) 
          LogW[n]\)\( | \_\(n = n\&_\)\)\) = \(\(-\(1\/N\)\) \((1\/p + \ 
              1\/q)\) = \(\(-\(1\/N\)\) \((\(q + p\)\/\(q\ p\))\) = \(-\(1\/\(N\ p\ q\)\)\)\)\)\)\)\)

Utilization of the two derivatives (17) and (11) in the Taylor expansion of Log W[n] equation (5) yields (keeping the first three terms)

\(\(\(\ \ \)\(Log\ W[n] = 
      Log\ W[n\&_]\  + \ 0\ ×\ \((n - n\&_)\) - 
        1\/\(N\ p\ q\)\ \((n - n\&_)\)\^2\/\(2!\)\)\)\)

Since the exponential and Log functions are inverses we can also write

\(\(\(\ \)\(W[n] = 
      W[n\&_]\ e\^\(-\(\((n - n\&_)\)\^2\/\(2\ N\ p\ q\)\)\)\)\)\)

which is the basic Gauss approximation to the Binomial distribution.  \(TraditionalForm\`n\&_\) =Np was called μ in the  Binomial distribution so

\(\(\(\ \)\(μ = n\&_\)\)\)

and we also introduce the quantity (σ^2) via

(σ^2)≡Npq  .

Then the Gauss distribution (19) appears

W[n]=A (e^(-(((((n - μ))^2))/(2(σ^2)))))

where we called A=\(W[n\&_]\) for simplicity and to note \(W[n\&_]\) is a constant independent of n.

The Normalization of the Gauss Distribution

The normalization condition for the Binomial distribution is

(∑_(n = 0)^N) W[n] =  1

Since for N large W[n] can be thought of as continuous, we write (20) as an integral

(∫_0^∞) W[n] dn  =  
      1

and substituton of the Gauss approximation (19) we also obtain

A (∫_0^∞) (e^(-(((((n - μ))^2))/(2(σ^2)))))dn = 1

where A was taken outside the integrand as it is constant independent of n.  The integral in equation (22) is standard and Mathematica yields (using σS as (σ^2)).

Clear[n, μ, σS]

A(∫_0^∞) (E^(-(((((n - μ))^2))/(2 σS))))\[DifferentialD]n

The result above appears complicated because of the Error Function Erf[x].  It not necessary to know the form of Erf[x] however, since we note that μ=Np and (σ^2)≡ 2  Npq  and by definition (21) so the ratio which the argument of Erf[x] appears

\(\(\(\ \)\(μ\/\@\(2\ σ\^2\) = 
      Np\/\@\(2  Npq\) ≈ \ \@\(p\/q\)\ ×\ \@N\  -> \ \(∞\ \ \ \ as\ \ \ N -> \(\(∞\)\(.\)\)\)\)\)\)

Mathematica also gives the limit

Erf[∞]

so the integral reduces to

\(\(\(\ \)\(\(∫\_0\%∞\ \(e\^\(-\(\((n - μ)\)\^2\/\(2  σ\^2\)\)\)\) dn\)\(=\)\(\ \)\(\@\(2\ π\ σ\^2\)\)\(\ \ \)\)\)\)

This integral can be obtained more easily by noting that the lower limit can be extended to -∞ since the integral is almost zero there (the Gauss distribution is peaked about μ). Thus Mathematica yields more directly (and simply) the result

(∫_(-∞)^∞) (E^(-(((((n - μ))^2))/(2 σS))))\[DifferentialD]n

Combining (27) with (25) yields

A (Sqrt[2 π (σ^2)])  =  1   or   A  =  
        (1/(Sqrt[2 π (σ^2)]))

so we know know the constant (called the normalization) in the Gauss distribution (22) which now appears

W[n]=(1/(Sqrt[2 π (σ^2)])) (e^(-(((((n - μ))^2))/(2 (σ^2)))))

We want to understand the other parameters in equation (29).

The Average for the Gauss Distribution

The average \(n\&_\) is by definition

\(\(\(\ \)\(n\&_ = ∫\_0\%∞ 
          n\ W[n] \(\(d\)\(n\)\(\ \)\)\)\)\)

and using the Gauss distribution (29)

\(n\&_\)=(1/(Sqrt[2 π (σ^2)]))(∫_0^∞)n   (e^(-(((((n - μ))^2))/(2 (σ^2)))))\(\(d\)\(n\)\(\ \)\)

Again the integral is performed with Mathematica (extending the integral to -∞) obtaining

(∫_(-∞)^∞)n   (E^(-(((((n - μ))^2))/(2 σS))))\[DifferentialD]n

Using this value for the integral in equation (31) yields

\(\(\(\ \ \)\(n\&_ = \(\(1\/\@\(2\ π\ σ\^2\)\) \@\(2\ π\)\ μ\ \@σ\^2 = μ\)\)\)\)

which is a result we suspected was true as the Gauss distribution comes from the Binomial distribution where this already was know to be true.

The Variance or Standard Deviation

The average of the squares of the deviations from the average or variance is defined by the integral

\(\(\((n - μ)\)\^2\)\&_ = \ \ ∫\_0\%∞\((n - μ)\)\^2\ W[n] \(\(d\)\(n\)\(\ \)\)\)

and using the Gauss distribution (29) for W[n] yields

\(\(\((n - μ)\)\^2\)\&_\)=(1/(Sqrt[2 π (σ^2)]))(∫_0^∞)(((n - μ))^2) (e^(-(((((n - μ))^2))/(2 (σ^2)))))\(\(d\)\(n\)\(\ \)\)

Mathematica can perform this integral (after extending the lower limit to -∞ to simplify the result)

(∫_(-∞)^∞)(((n - μ))^2) (E^(-((((n - μ))^2)/(2 σS)))) \[DifferentialD]n

So the average of the squares of the deviation from the average is

\(\(\(\ \)\(\(\((n - μ)\)\^2\)\&_ = \(\(1\/\@\(2\ π\ σ\^2\)\) \@\(2\ π\)\ σ\^3 = σ\^2\)\)\)\)

so we were entitled to use the symbol for the variance (σ^2)in definition (21) since as a result

\(\(\(\ \)\(σ\^2 = ∫\_0\%∞\((n - μ)\)\^2\ W[n] \(\(dn\)\(\ \ \)\(.\)\)\)\)\)

Error Function Integrals

A useful thing to calculate is the probability of the Gauss distribution in the region one standard deviation σ on either side of the average μ:

(∫_(μ - σ)^(μ + σ))W[
              n] dn=(1/(Sqrt[2 π (σ^2)]))(∫_(μ - σ)^(μ + σ))(e^(-(((((n - μ))^2))/(2 (σ^2)))))\(\(d\)\(n\)\(\ \)\)

Mathematica yields

(∫_(μ - σ)^(μ + σ))(E^(-(((((n - μ))^2))/(2 (σ^2)))))\[DifferentialD]n

Combining this result in equation (37) yields

\(\(\(\ \)\(∫\_\(μ - σ\)\%\(μ + σ\)W[
            n] dn = \(\(1\/\@\(2\ π\ σ\^2\)\) \@\(2\ π\)\ σ\ Erf[1\/\@2] = Erf[1\/\@2]\)\)\)\)

Mathematica provides the numerical value of this function via

Erf[(1/(Sqrt[2]))] // N

so we conclude

(∫_(μ - σ)^(μ + σ))W[
            n] dn = 0.68

for the Gauss distribution.  Thus 68% of the probablity is within one standard deviation σ of the average value μ.

One can consider the amount of probability within two standard deviations similarly

\(∫\_\(μ - 2\ σ\)\%\(μ + 2\ σ\)W[n] dn\)=(1/(Sqrt[2 π (σ^2)]))(∫_(μ - 2 σ)^(μ + 
                  2 σ))(e^(-(((((n - μ))^2))/(2 (σ^2)))))\(\(d\)\(n\)\(\ \)\)

and Mathematica yields

(∫_(μ - 2 σ)^(μ + 2 σ))(E^(-(((((n - μ))^2))/(2 (σ^2)))))\[DifferentialD]n

which when combined with equation (40) yields

\(\(\(\ \ \)\(∫\_\(μ - 2\ σ\)\%\(μ + 2\ σ\)W[n] dn = \(\(1\/\@\(2\ π\ σ\^2\)\) \@\(2\ π\)\ σ\ Erf[\@2] = \ Erf[\@2]\)\)\)\)

The numerical value of this Error function is

Erf[(Sqrt[2])] // N

so for the Gauss distribution

\(\(\(\ \)\(∫\_\(μ - 2\ σ\)\%\(μ + 2\ σ\)W[n] dn = 0.95\)\)\)

Thus 95% of the probablity is within two standard deviations σ of the average value μ.

Continuing further we can consider the amount of probability within three standard deviations similarly

\(∫\_\(μ - 3\ σ\)\%\(μ + 3\ σ\)W[n] dn\)=(1/(Sqrt[2 π (σ^2)]))(∫_(μ - 3 σ)^(μ + 
                  3 σ))(e^(-(((((n - μ))^2))/(2 (σ^2)))))\(\(d\)\(n\)\(\ \)\)

and Mathematica yields

(∫_(μ - 3 σ)^(μ + 3 σ))(E^(-(((((n - μ))^2))/(2 (σ^2)))))\[DifferentialD]n

which when combined with equation (43) yields

\(\(\(\ \ \)\(∫\_\(μ - 2\ σ\)\%\(μ + 2\ σ\)W[n] dn = \(\(1\/\@\(2\ π\ σ\^2\)\) \@\(2\ π\)\ σ\ Erf[3\/\@2] = \ Erf[3\/\@2]\)\)\)\)

The numerical value of this Error function is

Erf[(3/(Sqrt[2]))] // N

so for the Gauss distribution

\(\(\(\ \)\(∫\_\(μ - 2\ σ\)\%\(μ + 2\ σ\)W[n] dn = 0.997\)\)\)

Thus 99.7% of the probablity is within three standard deviations σ of the average value μ.  It is really unlikely that a random experimental error will result in a measured value greater than three standard deviations away from the average measured value.

©  Rodney L. Varley (2001).