Measurement uncertainty

From CEOpedia
Measurement uncertainty
See also


Measurement uncertainty - is a non-negative parameter characterizing the dispersion of the quantity values being attributed to a measurand, based on the information used. That is the wording of the International Vocabulary of Metrology (VIM), definition 2.26[1]. In plain terms it answers one question: after measuring something as carefully as we could, how far might the real value still be from the number we wrote down?

A length of 50.023 mm means little on its own. Written as 50.023 mm ± 0.022 mm, it tells a buyer or an auditor what the number is worth. The parameter is usually a standard deviation (the standard uncertainty) or the half-width of an interval with a stated coverage probability (the expanded uncertainty). Laboratories and factories evaluate it according to one international document, the Guide to the Expression of Uncertainty in Measurement, known simply as the GUM.

Background

For a long time there was no international consensus on how to state uncertainty, so results from different laboratories couldn't be compared properly.

In 1977 the Comité International des Poids et Mesures (CIPM) asked the BIPM to tackle the problem. A questionnaire went to 32 national metrology laboratories; 21 had replied by early 1979. Almost all wanted a common procedure. Which one, nobody agreed. A working group with experts from 11 national laboratories then drafted Recommendation INC-1 (1980), Expression of Experimental Uncertainties. The CIPM approved it in 1981 and reaffirmed it in 1986[2].

INC-1 was a short outline, not a manual. Detailed rules were written by an ISO working group with experts nominated by the BIPM, IEC, ISO and OIML. The Guide was published in October 1993 and reprinted with corrections in 1995[3]. In 2008 the Joint Committee for Guides in Metrology (JCGM) reissued it as JCGM 100:2008, which is the 1995 version with minor corrections and is free to download from the BIPM website[2]. A family of supplements followed, among them JCGM 101:2008 on the Monte Carlo method and JCGM 106:2012 on conformity assessment.

NIST's Technical Note 1297 by Taylor and Kuyatt condensed the approach into a short document for NIST staff. Still one of the easiest introductions[3].

Error versus uncertainty

The two words are often used as synonyms. They shouldn't be. Measurement error is a single number: the measured value minus the reference value. It has a sign, and in most real situations it is unknown, because the true value is unknown. Uncertainty is a statement about our state of knowledge. It has no sign, it can be evaluated without knowing the true value, and it describes a spread of plausible values rather than one deviation.

The GUM puts it bluntly: a corrected result can, unknowably, have negligible error and still a large uncertainty. And an unrecognized systematic effect can't enter the uncertainty evaluation at all, although it does contribute to the error[4].

Aspect Measurement error Measurement uncertainty
What it is Difference between a measured value and a reference (true) value Parameter describing the dispersion of values reasonably attributed to the measurand
Sign Positive or negative Always non-negative
Can it be known? Only when a reference value is available Evaluated from data and other information, even without a reference value
What to do with it Known systematic error is corrected Reported together with the result
Typical form e = +0.02 mm u = 0.011 mm, or U = 0.022 mm (k = 2)
Classification Systematic and random components Type A and Type B evaluation (by method, not by origin)

The last row matters. A component that is random in one measurement can become systematic in the next, when the first result is used as an input. So the GUM's labels A and B describe how a component was evaluated, not what kind of effect produced it[4].

Sources of uncertainty

GUM clause 3.3.2 lists the sources that turn up in practice[4]:

  • incomplete definition of the measurand (the "diameter" of a shaft that isn't perfectly round) and imperfect realization of that definition,
  • non-representative sampling, where the sample measured doesn't represent the defined measurand,
  • inadequate knowledge of environmental effects, or imperfect measurement of environmental conditions (temperature, humidity, pressure),
  • personal bias in reading analogue instruments,
  • finite instrument resolution or discrimination threshold,
  • inexact values of measurement standards, reference materials, constants and other external parameters,
  • approximations and assumptions built into the measurement method and procedure,
  • variations in repeated observations under apparently identical conditions.

These sources overlap: operator effects and environmental drift partly show up in the scatter of repeated readings. Count them twice and the result is inflated.

Type A and Type B evaluation

Type A

A Type A evaluation uses statistics on a series of observations. Take n independent readings q1, ..., qn of the same quantity. Their mean is the best estimate, and the experimental standard deviation of the readings is

\[ s(q) = \sqrt{\frac{1}{n-1}\sum_{k=1}^{n}\left(q_k-\bar{q}\right)^2} \]

What counts for the result, though, is the uncertainty of the mean, not of a single reading:

\[ u(\bar{q}) = \frac{s(q)}{\sqrt{n}} \]

Four times as many readings only halve it[5].

Type B

A Type B evaluation uses everything else: calibration certificates, manufacturer specifications, handbook data, previous measurements, experience with the instrument. The trick is to convert such information into a standard deviation.

  • If a certificate quotes an uncertainty as a multiple of a standard deviation (for example U with k = 2), the standard uncertainty is that value divided by the multiplier.
  • If only bounds ±a are known and any value between them is equally likely, the GUM assigns a rectangular (uniform) distribution:

\[ u = \frac{a}{\sqrt{3}} \]

  • If values near the centre are more likely than values near the limits, a triangular distribution is more realistic:

\[ u = \frac{a}{\sqrt{6}} \]

Digital resolution is the textbook rectangular case. A display that shows 0.01 mm steps hides anything within ±0.005 mm, so u = 0.005/√3 ≈ 0.0029 mm. The GUM itself observes that the variances of rectangular, triangular and normal assumptions are surprisingly similar for the same bounds[5]. Both types end up as standard deviations and are treated identically afterwards. Type B isn't "less scientific", just based on different evidence.

Combined standard uncertainty

Most measurands are calculated from input quantities through a model Y = f(X1, X2, ..., XN). Density from mass and volume. Power from voltage and current. Length from a reading plus a temperature correction. The combined standard uncertainty uc(y) comes from the law of propagation of uncertainty. For uncorrelated inputs it reads[6]:

\[ u_c^2(y) = \sum_{i=1}^{N} \left(\frac{\partial f}{\partial x_i}\right)^2 u^2(x_i) = \sum_{i=1}^{N} c_i^2\, u^2(x_i) \]

The partial derivatives ci are sensitivity coefficients: how strongly the output reacts to each input, the logic of sensitivity analysis applied to a measurement. When inputs are correlated (two resistors calibrated against the same standard, say), covariance terms are added.

Squares add. That has a practical consequence people often overlook: the largest component dominates. A contribution one third the size of the biggest one adds only about 5% to the total. Effort spent shrinking small components is mostly wasted.

The formula is a first-order Taylor approximation: exact for linear models, unreliable for strongly non-linear ones.

Expanded uncertainty and coverage factor

A standard uncertainty corresponds to roughly one standard deviation. Customers usually want a wider interval, one expected to contain most of the values that could reasonably be attributed to the measurand. The expanded uncertainty is

\[ U = k \cdot u_c(y) \]

where k is the coverage factor. The GUM says k will generally be between 2 and 3. If the output distribution is approximately normal and the effective degrees of freedom are not too small, k = 2 gives an interval with a level of confidence of about 95%, and k = 3 about 99%[6]. When the Type A part rests on only a few readings and dominates the budget, k is taken from the t-distribution with effective degrees of freedom (the Welch-Satterthwaite formula in GUM Annex G) and ends up larger than 2.

Reporting the result

The GUM gives worked wording for reports. Its example in clause 7.2.4 is a mass standard:

mS = (100.021 47 ± 0.000 79) g, where the number after ± is an expanded uncertainty U = k uc, with uc = 0.35 mg and k = 2.26 based on the t-distribution for 9 degrees of freedom, defining an interval estimated to have a level of confidence of 95 percent[6].

The rules behind the example are simple:

  • state the coverage factor (or the coverage probability) every time,
  • give the uncertainty with no more than two significant digits,
  • don't give the result with more digits than the uncertainty justifies,
  • say what the uncertainty includes and, if relevant, what it leaves out.

A bare "± 0.02" without k is ambiguous.

Uncertainty budget

An uncertainty budget is the table listing all components: source, value, distribution, divisor, standard uncertainty, sensitivity coefficient and contribution. A good budget shows at a glance which component dominates and where improvement would pay off. It also makes assumptions visible, which is half the point.

Analytical chemists often fill the budget differently. The EURACHEM/CITAC guide Quantifying Uncertainty in Analytical Measurement shows how method validation data, internal quality control results and proficiency-test data can be used to estimate uncertainty, instead of modelling every weighing and dilution step separately[7]. The two routes are often called "bottom-up" and "top-down".

Monte Carlo method

JCGM 101:2008, the first supplement to the GUM, replaces the propagation of uncertainties with the propagation of distributions. Each input quantity gets a probability density function. A computer draws random values from all of them, pushes each set through the measurement model, and repeats this M times. The resulting cloud of output values approximates the distribution of the measurand; its standard deviation is the standard uncertainty, and its percentiles give a coverage interval directly. No derivatives, no coverage factor.

The supplement names two cases where the method pays off: when linearization of the model is an inadequate representation, and when the output distribution departs appreciably from a Gaussian or scaled t-distribution, for instance through marked asymmetry. It notes that M = 106 trials can often be expected to deliver a 95% coverage interval whose length is correct to one or two significant decimal digits[8]. The technique is ordinary Monte Carlo simulation applied to a measurement model, and it is often used just to check whether the simpler GUM framework is adequate.

Decision rules and conformity assessment

Measurements are rarely an end in themselves. Is the part within tolerance? Does the batch meet the legal limit? Uncertainty turns such yes/no questions into questions about risk.

Take a measured value just below the upper tolerance limit. With uncertainty ±U around it, there's a fair chance the true value is above the limit. Accepting the item means accepting a probability of false acceptance (consumer's risk); rejecting it creates a risk of false rejection (producer's risk). JCGM 106:2012 treats this formally, using probability density functions for the measurand and distinguishing the specific risk for one item from the global risk across a population of items[9].

ISO/IEC 17025:2017, the standard against which testing and calibration laboratories obtain accreditation, requires laboratories to evaluate measurement uncertainty (clause 7.6). When a customer requests a statement of conformity, the specification and the decision rule must be clearly defined and agreed with the customer (clause 7.1.3). Clause 7.8.6 requires the laboratory to document the decision rule, taking into account the level of risk (false accept, false reject and statistical assumptions), and to state in the report which decision rule was applied[10].

The usual tool is a guard band w. The acceptance limit is moved inside the tolerance limit, AL = TL − w, with w typically a multiple of U. ILAC-G8:09/2019 tabulates common choices for a single-sided limit and normally distributed results[10]:

Decision rule Guard band w Specific risk
Simple acceptance 0 probability of false accept < 50%
ISO 14253-1:2017 0.83 U PFA < 5%
ILAC G8:2009 rule 1 U PFA < 2.5%
"3 sigma" 1.5 U PFA < 0.16%
"6 sigma" 3 U PFA < 1 ppm

Wider guard bands protect the customer and cost the producer, since more conforming items get rejected. Where to draw the line is a business decision, not a metrological one. The same logic sits behind quality inspection and acceptance sampling, and it is why measurement system analysis and process capability studies demand a measurement system whose variation is small relative to the tolerance.

Example of measurement uncertainty

The following example is hypothetical; numbers are chosen for illustration.

An aluminium part with a nominal length of 50 mm is measured with a digital calliper (resolution 0.01 mm) in a workshop. The calibration certificate of the calliper states an expanded uncertainty U = 0.02 mm with k = 2. The workshop temperature is only known to lie within 20 ± 3 °C, and the steel calliper and aluminium part expand differently: taking about 23 × 10-6/°C for aluminium and 11.5 × 10-6/°C for steel, the difference is 11.5 × 10-6/°C.

Ten readings (mm): 50.02, 50.03, 50.01, 50.02, 50.04, 50.02, 50.03, 50.01, 50.02, 50.03.

  • Sum = 500.23 mm, mean = 50.023 mm.
  • Sum of squared deviations = 0.000 81 mm², so s = √(0.000 81 / 9) = 0.0095 mm.
  • Type A standard uncertainty of the mean: 0.0095 / √10 = 0.0030 mm.
  • Resolution: half-width 0.005 mm, rectangular, 0.005 / √3 = 0.0029 mm.
  • Calibration: 0.02 / 2 = 0.0100 mm.
  • Temperature: maximum effect 50 mm × 11.5 × 10-6/°C × 3 °C = 0.001 725 mm, rectangular, 0.001 725 / √3 = 0.0010 mm.

All inputs enter the model additively, so every sensitivity coefficient equals 1.

Source Type Value / half-width (mm) Distribution Divisor Standard uncertainty u (mm) Share of uc²
Repeatability (10 readings) A s = 0.0095 normal √10 0.0030 7.6%
Resolution of display B 0.005 rectangular √3 0.0029 7.0%
Calibration certificate B U = 0.02 (k = 2) normal 2 0.0100 84.5%
Temperature difference B 0.001 725 rectangular √3 0.0010 0.8%
Combined 0.0109 100%
Expanded (k = 2) 0.022

\[ u_c = \sqrt{0.0030^2 + 0.0029^2 + 0.0100^2 + 0.0010^2} \approx 0.0109 \text{ mm} \]

\[ U = 2 \times 0.0109 \approx 0.022 \text{ mm} \]

Result: L = (50.023 ± 0.022) mm, where the number after ± is an expanded uncertainty with coverage factor k = 2, corresponding to a coverage probability of approximately 95%.

What does the budget say? Calibration of the calliper accounts for about 85% of the combined variance. Taking 100 readings instead of 10 would cut the Type A part to under 0.001 mm and move U only from 0.022 to about 0.021 mm. Pointless. A better-calibrated or better instrument (a micrometer) is the only real lever. Temperature is negligible here, though it wouldn't be for a 2 m part.

Now suppose the drawing says 50.00 ± 0.05 mm, so the upper tolerance limit is 50.05 mm. With a guard band w = U, the acceptance limit becomes 50.05 − 0.022 = 50.028 mm. The part at 50.023 mm passes. A second part reading 50.035 mm would pass under simple acceptance, yet would be rejected under the guard-band rule, because its true length could plausibly exceed 50.05 mm.

Advantages of measurement uncertainty

  • Results from different laboratories, countries and years become comparable, because everyone states uncertainty the same way.
  • Pass/fail decisions carry a known risk instead of an implicit one, which supports quality control, quality assurance and risk management.
  • The budget shows where money spent on better equipment or training actually reduces uncertainty, in the spirit of reduction of uncertainty as a management goal.
  • It forces explicit documentation of assumptions, which helps audits under ISO/IEC 17025 and the measurement-resource requirements of ISO 9001.
  • Known biases get corrected instead of hidden inside a wide tolerance.

Limitations and criticisms

  • Assumptions of linearity and normality. The law of propagation is a first-order approximation, and k = 2 ≈ 95% presumes an approximately normal output. Skewed or bounded quantities break both; the Monte Carlo method fixes this but requires software and modelling skill.
  • Subjectivity of Type B. Choosing a distribution, a half-width or a correlation is a judgement. Two competent analysts can arrive at different budgets for the same measurement.
  • Mixed philosophy. Type A rests on frequentist statistics, Type B on degrees of belief, and the GUM puts both into one formula. JCGM 101 instead assigns probability distributions to all inputs using Bayes' theorem and the principle of maximum entropy.
  • Unknown unknowns. An unrecognized systematic effect never enters the budget. A small uncertainty is no guarantee of a small error.
  • Cost. Building and maintaining budgets for hundreds of methods takes time, training and calibration money.
  • Misuse. Uncertainty gets quoted without k, or copied from a calibration certificate as if it were the uncertainty of the user's own measurement.
  • Dependence on the model. A wrong model yields a precise-looking but wrong budget. Statistical process control of check standards and interlaboratory comparisons show whether the stated uncertainty is realistic.

References

Footnotes

  1. ↑ JCGM 200:2012, International vocabulary of metrology (VIM), 3rd edition, 2012, definition 2.26
  2. ↑ JCGM 100:2008, Guide to the expression of uncertainty in measurement (GUM), 2008, Foreword and Introduction
  3. ↑ Taylor B.N., Kuyatt C.E., Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results, NIST Technical Note 1297, 1994, Preface
  4. ↑ JCGM 100:2008, Guide to the expression of uncertainty in measurement (GUM), 2008, clauses 3.3.1-3.3.4
  5. ↑ JCGM 100:2008, Guide to the expression of uncertainty in measurement (GUM), 2008, clauses 4.2-4.3 (4.3.3, 4.3.7, 4.3.9)
  6. ↑ JCGM 100:2008, Guide to the expression of uncertainty in measurement (GUM), 2008, clauses 5.1.2, 6.3.1, 6.3.3, 7.2.4 and 7.2.6
  7. ↑ Ellison S.L.R., Williams A. (eds.), EURACHEM/CITAC Guide CG 4: Quantifying Uncertainty in Analytical Measurement, 3rd edition, 2012
  8. ↑ JCGM 101:2008, Supplement 1 to the GUM - Propagation of distributions using a Monte Carlo method, 2008, clause 1 (Scope), clauses 6 and 7.2
  9. ↑ JCGM 106:2012, The role of measurement uncertainty in conformity assessment, 2012
  10. ↑ ILAC-G8:09/2019, Guidelines on Decision Rules and Statements of Conformity, 2019, section 2 and Table 1
Measurement uncertainty — recommended articles
Measurement error — Measurement system analysis — Systematic error — Random error — Monte Carlo simulation — Process capability — Quality control
Retrieved from CEOpedia