Statistical Outlier Treatment in a Water Quality Monitoring Dataset: Rosner's Test, Worked

Twenty-eight quarterly chloride results pooled from four upgradient wells across seven quarters of detection monitoring, and one of them reads 188 mg/L against a background that otherwise runs 25–46. The background prediction limit you compute from that pool is the number every downgradient well gets compared against for the next several years, so statistical outlier treatment in a water quality monitoring dataset is not a tidiness exercise — leaving the 188 in doubles the limit and buys a real release several years of invisibility. This works the arithmetic end to end on that dataset, using the test EPA’s own software applies, and then covers the part practitioners get wrong more often than the math: what you are allowed to do once the test fires.

What Statistical Outlier Treatment Decides in a Water Quality Monitoring Dataset

An outlier test answers one narrow question: is this value improbable enough, under a normal distribution fitted to the rest of the data, that it probably did not come from the same population? It does not tell you the value is wrong, and it does not authorize deletion. ProUCL 5.1’s technical guide puts extreme outliers into two categories: low-probability observations from the far tail of the main population, or observations from a genuinely different population. It is equally direct about the limits of the test itself — “the statistical tests alone cannot determine whether a statistical outlier should be investigated further.”

Which statistical procedures a program may use is not always your choice, either. RCRA groundwater monitoring at permitted facilities works under the general requirements at 40 CFR 264.97, and permits and cleanup orders routinely name the method. Run the test to inform the decision; check the governing document before you change what the compliance number is built from.

For an upgradient background pool, the second case is the one that matters. A 188 mg/L chloride result in a road-salt-influenced aquifer may be a perfectly real measurement of a different population than the one you are trying to characterize. That still disqualifies it from the background pool, and for a different reason than “it is an error.”

Which Test Applies: Dixon or Rosner

ProUCL implements two classical tests, and the sample size picks between them. Dixon’s extreme value test applies when n ≤ 25; Rosner’s test applies at n ≥ 25 and can identify up to 10 outliers. Both assume the data without the suspected outliers are normally distributed, and both suffer from masking — the presence of several outliers can hide the fact that even one is present.

Masking is why the choice matters. Dixon in the form EPA guidance presents tests a single suspect, so a second extreme value inflates the denominator and can push the ratio below its critical value. Rosner sidesteps this by removing suspects one at a time and recomputing, which is why it is the right test whenever you have enough data to use it. One detail worth knowing: the Rosner implementation in EPA’s ProUCL software uses the generalized extreme studentized deviate, while the form in older EPA guidance uses the standard ESD. A hand calculation from a textbook table will not always reproduce ProUCL exactly.

The Formula

At each step i, compute the test statistic from the working dataset — the full set with the previously removed points deleted:

In plain terms: the distance from the current mean to the most extreme remaining value, in current standard deviations. The critical value it is compared against is

where n is the size of the original dataset, t is the Student’s t critical value at probability p with n − i − 1 degrees of freedom, and α is the significance level (0.05 here). The Bonferroni-style adjustment inside p is what keeps the family-wise error rate honest across repeated tests on the same data.

Worked Example: 28 Background Chloride Results

The pooled upgradient dataset, in mg/L, sample order:

31.2, 27.8, 34.5, 29.1, 41.3, 36.0, 25.4, 38.7, 30.6, 33.2, 28.9, 44.1, 26.7, 35.8, 39.4, 32.5, 37.1, 29.8, 42.6, 24.9, 31.9, 46.2, 34.0, 27.3, 188.0, 33.7, 40.5, 30.1

Note how far the second step falls from significance. That gap is the useful diagnostic. A dataset where R₂ lands just under λ₂ is telling you the pool may contain a second population rather than a single stray value, and it is worth a Q-Q plot before you accept the result.

The Consequence: What One Value Does to the Prediction Limit

The reason to run the test at all is the number it changes downstream. A 95% upper prediction limit for one future observation is

With the 188 included: 39.332 + (1.7033 × 29.697 × 1.0177) = 90.8 mg/L. With it excluded: 33.826 + (1.7056 × 5.854 × 1.0184) = 44.0 mg/L.

A downgradient well at 60 mg/L clears the first limit comfortably and trips the second. One retained value is the difference between detecting that release and not detecting it, and the effect runs in the direction that hurts — toward false negatives, in a program whose entire purpose is early detection. It compounds too: an inflated background limit stays in place across subsequent monitoring events until somebody recomputes it. The same sensitivity applies to Mann-Kendall trend testing on compliance monitoring data, though less severely, because a rank-based test is far less sensitive to the magnitude of an extreme value than a mean-and-variance statistic is.

The Normality Trap

Both tests assume normality, and practitioners routinely test the wrong dataset for it. ProUCL addresses this directly: “users tend to believe that the original data (with outliers) should follow a normal distribution. A data set with outliers very seldom follow a normal distribution as the presence of outliers tends to destroy the normality of a data set.” The assumption applies to the main body of the data, after the suspects are removed.

This dataset makes the point cleanly. Shapiro-Wilk on all 28 values gives W = 0.355, p < 0.001 — a decisive rejection of normality. Drop the 188 and the same test on the remaining 27 gives W = 0.965, p = 0.48. The distribution was normal all along; one value was hiding it.

What to Do After the Test Fires

The test result is an input to a decision, not the decision. In practice:

1. Look at the data graphically. ProUCL recommends pairing every outlier test with Q-Q plots and box plots, which reveal mixture populations that a test statistic cannot.
2. Go back to the source. Pull the lab package for that sample — a dilution, a transcription error, or a preservation problem shows up in the paperwork. Check the field notes and the well’s purge record too.
3. Run the statistics both ways. This is ProUCL’s stated recommendation: compute with and without the outlier and evaluate the impact, rather than picking one version and reporting it alone.
4. Write down the disposition and why. “Excluded from the background pool as a road-salt-influenced result inconsistent with the upgradient population, per Rosner’s test at α = 0.05” survives regulatory review. “Removed as an outlier” does not.

Point 4 is where most of the actual risk sits. An excluded value that reappears in the lab data but not in the statistical summary, with no documented basis, is the kind of gap that gets a background threshold thrown out. If the dataset also carries non-detects, note that the disposition interacts with censoring — the methods for handling non-detects and censored statistics assume a defined population, and changing which values belong to that population changes those estimates too. Keep the disposition record with the dataset, not in a separate memo, so it travels with the numbers through the rest of the groundwater monitoring data workflow.

Statistical methods for background comparison are program-specific and the applicable approach is set by your permit, order, or state guidance; the arithmetic above is a starting point for scoping the analysis rather than a substitute for the criteria your program imposes. Other posts in the water quality compliance archive cover the reporting steps downstream of the statistics.