3. Least Square Fit to a Straight Line

Straight line (or linear) data often results from experiments.  There is a mathematical technique for finding the "best" straight line that fits your experimental data points.

The Experimental Data

Suppose you have N experimental data points in a Table

data=({{(Subscript[x, 1]), (Subscript[y, 1])}, {(Subscript[x, 2]), (Subscript[y, 2])}, {(Subscript[x, 3]), (Subscript[y, 3])}, {..., ...}, {(Subscript[x, N]), (Subscript[y, N])}})

Alternatively, you could input the data a list

data = {{(Subscript[x, 1]), (Subscript[y, 1])}, {(Subscript[x, 2]), (Subscript[y, 2])}, {(Subscript[x, 3]), (Subscript[y, 3])},  ... ,  {(Subscript[x, N]), 
          (Subscript[y, N])}}

As a concrete example, consider the pressure and temperature data for an ideal gas (at constant volume).  The temperature might be regarded as the dependent variable (graphed on the vertical axis) and then the pressure is the independent variable (graphed on the horizontal axis).  Which is the independent variable is determined by the experimental situation and usually the independent variable is the physical quantity you control directly.  Changes in the independent variable cause changes in the dependent variable.  Sometimes which is dependent and independent in the experimental situation is not obvious and the choice of which variable is graphed on the horizontal axis is arbitrary in these cases.

Some specific pressure and temperature data

data=({{65, -20. }, {75, 17}, {85, 42}, {95, 94}, {105., 127}})

The data in the first column (of the table) is the pressure in mm and the data in the second column is the corresponding temperature in °C.  A graph of the data appears

\(ListPlot[data, 
      AxesLabel -> {"Pressure", "Temperature"}, 
PlotStyle -> PointSize[0.02]]\)

where we labeled the axes and increased to data point size to make it more visible.

The data appears to lie along a straight line.  Sometimes it is adequate to just "eyeball" or guess at the best straight line.  When you "eyeball" what you are actually doing is drawing a straight line so that equal numbers of data points appear above the line as below.  For example, an "eyeball" straight line might appear

There are two data points above the line and three data points below the line.  Obviously where you draw the line is a bit subjective and it is better to have a definite mathematical procedure instead.

Regardless of how you get it from the data, the form of this "best" straight line would appear

y =  m x  +  b

where as m is the slope of the "best" straight line and b is the intercept.  We do not actually know the value of the slope m and intercept b yet.   In fact, the problem to be solved (by the method of Least Square Fitting) is the determination of m and b.  Right now it is just important to know that m and b exist (have definite numerical values).

Any given data point {(Subscript[x, k]), (Subscript[y, k])} is NOT on the "best" straight line.  The vertical distance between the data point and the point on the best straight line directly above the data point is called the "deviation".  Consider the third data point {(Subscript[x, 3]), (Subscript[y, 3])} on the graph above.  The deviation of the third data  point  is the distance indicated below:

The y value of the point on the line directly above the third data point is y[(Subscript[x, 3])] = m (Subscript[x, 3])  +  b and the "deviation" of the third data point is

deviation =  (Subscript[y, 3]) - 
          y[(Subscript[x, 3])]  =     (Subscript[y, 3]) - (m (Subscript[x, 3])  +  
              b)

Another way (procedure) of drawing the best straight line would be to minimize the deviations of all the data points from the "best" straight line.  Using this idea we would sum the deviations (like equation (2)) for all the data points.  Unfortunately, such a procedure does not work especially well as the deviation of a point above the line could cancel the deviation of a data point below the line (even though the best line does not pass through either data point).  
	A better procedure would be to square the deviations (like equation (2)) and sum all the squares of the deviations.  In this way, the deviations of data points would not cancel because only positive terms would be summed.  This sum of squares is called (σ^2) thus

\(\(\(\ \)\(σ\^2 = \ \(∑\+\(n = 1\)\%N\ \((y\_n - m\ x\_n\  - \ b)\)\^2\)\/N\)\)\)

σ is the Greek letter Sigma and (σ^2) is "Sigma-squared".   (σ^2) is  the average of the sum of the squares of the deviations from  the best straight line. Sometimes the "Least Square Fit" is called the "Sigma-squared Fit".  We want to find m and b such that (σ^2) is as small as possible.  This is what we mean by the "best" straight line for the given data points.  (The notation (σ^2) is tradition but we could have called this any function f.)  Note we could have alternatively called the "deviation" the shortest HORIZONTAL (instead of vertical) distance to the "best" straight line.  Doing this would cause problems in the derivation below.  However, the same things would result by just reversing the way the data is graph.

Calculus Problem:  Minimization

We have a function σ of two variables m and b so in general,  σ  =  σ[m,  b].  The graph of the function σ versus m and b is "surface" in three dimensions.    The problem is to find the value(s) of m and b so that σ is a minimum (or a valley on the surface representing σ).  This is a problem which is easily by using calculus.  
	If we require

\(\(\(\ \)\(∂\ σ\^2\/∂\ b = \ \(\(0\ \ \ and\ \ \ \ ∂\ σ\^2\/∂\ m\)\(=\)\(\ \)\(0\)\(\ \ \ \)\)\)\)\)

then where both of these conditions are met the surface has an extremum (either a minimum or a maximum). (Physically, we are looking for a minimum so it is usually not necessary to determine if the second derivatives are negative.)

Application of the Minimization Procedure:

Using the first of equation (4) and equation (3) yields

\(\(\(\ \)\(∂\ σ\^2\/∂\ b = \ \ \(\(\(\(∂\)\(\ \)\)\/∂\ 
                b\) \(∑\+\(n = 1\)\%N\ \((y\_n - m\ x\_n\  - \ b)\)\^2\) = 0\)\)\)\)

Calculating the derivatives is straight forward (if you have had third semester Calculus)

∂ ((σ^2)/∂) b =   - 
            2 (∑_(n = 1)^N) ((Subscript[y, n]) - m (Subscript[x, n])  -  b) = 
        0

Rearranging slightly we obtain

(∑_(n = 1)^N) (Subscript[y, n]) = 
      m (∑_(n = 1)^N)  (Subscript[x, n]) + b N

The last term results from the observation that

(∑_(n = 1)^N) 1 = N

Using the second of equation (4) and equation (3) yields

\(\(\(\ \)\(∂\ σ\^2\/∂\ m = \ \ \(\(\(\(∂\)\(\ \)\)\/∂\ 
                m\) \(∑\+\(n = 1\)\%N\ \((y\_n - m\ x\_n\  - \ b)\)\^2\) = 0\)\)\)\)

Calculating the derivatives is again  straight forward

∂ ((σ^2)/∂) m =   - 
            2 (∑_(n = 1)^N) ((Subscript[y, n]) - m (Subscript[x, n])  -  b) (Subscript[x, n]) = 
        0

Rearranging slightly we obtain

(∑_(n = 1)^N) (Subscript[y, n]) (Subscript[x, n]) = 
      m (∑_(n = 1)^N) ((Subscript[x, n])^2) + 
        b (∑_(n = 1)^N) (Subscript[x, n])

Equations (7) and (10) represent two equations for the two unknowns: m and b.

Solution of Two Simultaneous Equations

Equations (7) and (10) can be written compactly using matrix notation as

(({{(∑_(n = 1)^N) (Subscript[y, n])}, {(∑_(n = 1)^N) (Subscript[y, n]) (Subscript[x, n])}}))= (({{(∑_(n = 1)^N)  (Subscript[x, n]), N}, {m (∑_(n = 1)^N) ((Subscript[x, n])^2), (∑_(n = 1)^N) 
                (Subscript[x, n])}}))(({{m}, {b}}))

The solution to equations (7) and (10) for m and b can be obtain by elimination (or Cramer's Rule if you prefer to use matrices).  In either case, one obtains

\(m = \(N\ \ \(∑\+\(n = 1\)\%N\ \(y\_n\) x\_n\) - ∑\+\(n = 1\)\%N\ \ x\_n\ \(∑\+\(n = 1\)\%N\ y\_n\)\)\/Δ\)

\(\(\(\ \)\(b = \(∑\+\(n = 1\)\%N\ y\_n\ \ \(∑\+\(n = 1\)\%N\ x\_n\^2\) - ∑\+\(n = 1\)\%N x\_n\ \(∑\+\(n = 1\)\%N\ \(y\_n\) x\_n\)\)\/Δ\)\)\)

where the denominator Δ is given by

\(\(\(\ \)\(Δ\  = \ 
      N\ \(∑\+\(n = 1\)\%N\ 
            x\_n\^2\) - \(\(\ \)\(∑\)\)\+\(n = 1\)\%N\ \(\(x\_n\) \(∑\+\(n = 1\)\%N\ x\_n\)\)\)\)\)

Homework

Derive equations (11) and (12).  HINT:  First solve equation (7) for b.  Then use this equation to eliminate b in equation (10) obtaining an equation for m alone (without b).  Finally solve this equation for m and obtain equation (11) above. Finally use this value of m to find b.

Application to the Ideal Gas Data

We apply equation (11)-(13) to the ideal gas data presented previously and see how the Method of Least Square Fit finds the best straight line.  The pressure is the independent variable (horizontal axis) so we call these coordinate x.  The temperature then will be the y variables.  We input the x, y data points using

x[1] = 65. ; y[1] = -20. ;
x[2] = 75. ; y[2] = 17. ;
x[3] = 85. ; y[3] = 42. ;
x[4] = 95. ; y[4] = 94. ;
x[5] = 105;y[5] = 
      127. ;

The Table function can be used to construct the "data" label

data = Table[{x[n], y[n]}, {n, 1, 5}];

and the data is graphed using the function ListPlot

ExpGraph = ListPlot[data, PlotStyle -> PointSize[0.02]]

So we get a graph similar to what we got before.

The slope m is calculated using Num=5 (the number of data points)

Num = 5;
 Δ  =  
        Num*(∑_(n = 1)^Num) 
              x[n]*x[n] - (∑_(k = 1)^Num) x[
                k]*(∑_(m = 1)^Num) x[m];

m = ((Num*  (∑_(n = 1)^Num) y[n]*x[n] - (∑_(n = 1)^Num)  x[n] (∑_(n = 1)^Num) y[n])/Δ)

IMPORTANT NOTES:  The upper limit of summation is Num (instead of N) since N is reserved by Mathematica for the Numerical Approximation Function.  Also we inserted the star symbol for multiplication to make absolutely sure the we (and Mathematica) know exactly how the calculation is done in the above.

The intercept b is calculated using

b = (1/Δ) ((∑_(n = 1)^Num) 
              y[n]*  (∑_(n = 1)^Num) 
                  x[n]*x[n] - (∑_(n = 1)^Num) x[
                  n]* (∑_(n = 1)^Num) 
                  y[n]*x[n])

A graph of the "best" straight line appears

TheoGraph = Plot[m*P + b, {P, 65, 105}]

Note the pressure variable is from P=65 to P=105 which is the experimental range over which the data was taken.

Show[ExpGraph, TheoGraph]

From the figure, it is clear the "best" fit line agrees well with the data.  As an aside, when you took the introductory lab you extrapolated the theory backward (small P values) until the pressure P=0.  We thus extend the "best fit line" above using

TheoGraph = Plot[m*P + b, {P, 0, 105}]

so that now the combined graphs appears

Show[ExpGraph, TheoGraph]

The temperature T=T[P] which has zero pressure is determined (numerically) by

T[P_] := m*P + b

T[0]

and this is in qualitative agreement with the intercept in the figure above.

This intercept is defined as the "absolute zero" of temperature on the Centigrade scale.  (Absolute zero is the temperature for which the pressure of an ideal gas is zero.)  The actual value is a bit different from the above (which used actual student data) and instead is approximately -273 °C.

Using the Mathematica Function:  Fit[...]

The Mathematica function Fit[...] by also be used to determine the best straight line.

Fit[data, {1, x}, x]

This obviously is easier to use than equations (11)-(13) above.  More information can be obtained about the function Fit[...] using

?Fit

"data" obviously is the list of data points while x is the independent (horizontal) variable.  funs is a bit problematic as this is a list of variables you EXPECT will fit the data.  If you expect straight line (linear) data, then the list of functions is a constant (e.g. one) and a linear function (e.g.) so funs is just {1,x}.  Unfortunately, there are experiments where at the beginning you have no idea which functions are appropriate.  In cases like this, a more complicated procedure must be followed.

You obviously will probably use the function Fit to calculate the best straight line (instead of using equations (11)-(13)) after you leave this course (in so-called real life).  Here you will be asked to use equations (11)-(13) so to make sure you understand the idea behind the procedure.

© Rodney L. Varley (2005)