Detecting Gradual Drift

A temperature sensor reads 70°F, plus or minus a couple of degrees of noise. Nothing looks wrong on any single reading. But over hundreds of samples, the signal has drifted several degrees from its true setpoint — invisible to threshold alarms, buried in noise.

This recipe uses a fast/slow esMean (exponentially smoothed mean) crossover to surface the drift, a Page-Hinkley  change-point test to accumulate statistical evidence, and a persistence check to confirm the detection is sustained.

flow('drift-detector')
    .sanitize('sane', 'temperature', { failureReason: 'failReason' },
        { ranges: { temperature: { min: 0, max: 200 } } })
    .median3('m3', 'temperature', { median3: 'm3' })
    .esMean('fast', 'm3', { mean: 'fastMean' }, { halfLife: 1.35 })
    .esMean('slow', 'm3', { mean: 'slowMean' }, { halfLife: 13.5 })
    .diff('divergence', 'fastMean', 'slowMean', { diff: 'drift' })
    .pageHinkley('ph', 'drift',
        { phShift: 'driftDetected', phTestStatistic: 'phStat' },
        { delta: 0.1, lambda: 3 })
    .persistenceCheck('confirm',
        msg => msg.driftDetected === true,
        { persistenceConfirmed: 'confirmed' },
        { minVotes: 3, outOfTotal: 5 })
    .run()

Drag the slider and watch the fast and slow averages diverge as drift builds.

What You’re Seeing

The gray line is the raw temperature — noisy readings with a drift that starts at sample 200 but is invisible to the eye among the noise.

The cyan line (fast esMean, halfLife=1.35) responds to recent values within a few samples. The purple dashed line (slow esMean, halfLife=13.5) represents the long-term baseline — it takes roughly 10x longer to move.

During stable operation, both averages overlap. After sample 200, the fast esMean climbs with the drift while the slow esMean lags behind. Their divergence feeds the Page-Hinkley test, which accumulates evidence until the persistence check confirms. The rose vertical marks the confirmed detection.

Where This Pattern Fits

How It Works

Two esMean nodes with different half-lives create a fast/slow crossover detector. The fast esMean (halfLife=1.35, α≈0.40) responds to recent changes within a few samples. The slow esMean (halfLife=13.5, α≈0.05) represents the long-term baseline.

The diff node subtracts slow from fast. During stable operation, this difference fluctuates around zero. When drift begins, the fast esMean tracks the new level while the slow esMean lags, producing a sustained positive difference.

The Page-Hinkley test accumulates this difference as statistical evidence. The delta parameter (0.1) sets the minimum magnitude worth tracking; lambda (3) sets the evidence threshold for declaring a change. The persistence check (3 of 5) confirms the detection is sustained — not a transient spike in the test statistic.

esMean-based detection is robust to non-Gaussian noise — the ARL (Average Run Length before false alarm) drops only 5% under moderate skewness, compared to 56% for CUSUM (Lucas & Saccucci, 1990). This matters for real sensor data.

References

• Page, E.S. (1954). Continuous inspection schemes. Biometrika, 41(1/2), 100–115. doi:10.2307/2333009 
• Hinkley, D.V. (1971). Inference about the change-point from cumulative sum tests. Biometrika, 58(3), 509–523. doi:10.1093/biomet/58.3.509 
• Lucas, J.M. & Saccucci, M.S. (1990). Exponentially weighted moving average control schemes: properties and enhancements. Technometrics, 32(1), 1–12. doi:10.1080/00401706.1990.10484583 

Next Steps

• Detecting Sudden Shifts — the complementary recipe for abrupt step changes using Kalman innovation gating
• Detecting Sensor Freeze — detect when a sensor stops changing using running standard deviation collapse
• Under the Hood — understand what happens inside the pipeline