The Coefficient of Determination vs Relative Standard Error
The coefficient of determination, or R2 value, is a measure of how well a set of data fits a calibration curve. This is the metric that is used almost universally by agricultural and environmental laboratories across the county. The closer this value approaches 1, the better a calibration curve fits the range of standards.
This metric, however, isn’t without its shortcomings, especially in the lower concentration range. Consider the formula for the coefficient of determination below:
As you can see, the larger the error, the further away from one R2 becomes. Since at low concentration levels, the errors are relatively low compared to the higher range errors, minimizing error at higher concentration levels will yield an overall better R2 value. In essence, the higher-level standards are more heavily weighted, and as a result the R2 metric can have a value close to one despite large relative deviations in the response at low concentrations.
For this reason, many professionals in the laboratory community wish to see the relative standard error (RSE) become the new metric for determining the validity of a curve. This is because the RSE is much more sensitive to deviations from the calibration curve at both high and low concentrations. This is possible because the sum of square errors in the formula is normalized by the true value. See the RSE formula below. Consider the following example. Calibration 2 differs from Calibration 1 in that the red data point was decreased by a factor of eight.
From this example, it is evident that the R2 value is much less sensitive to an 8-fold decrease in response. The R2 value was barely affected while the RSE increased by an order of magnitude. This illustrates that you can have a high R2 value with substantial deviation at low concentrations. Conversely, the RSE is very sensitive to a change in response at any concentration and in this case, would deem calibration 2 unsatisfactory.
Why then, is the coefficient of determination still widely used? Old habits die hard. Also, it is quite easy to distinguish between a good and bad R2 value; i.e., the more lines the better. For the RSE, the lower the figure the better, but how low is dependent on the specified method. The coefficient of determination will continue to be used as it is a robust figure that works well for many applications, but as EPA regulations become more stringent, there may be a shift to using RSE.
