Detecting Sudden Shifts

A tank level reads 850 litres every minute. Then it reads 680. Is the sensor wrong, or did something just happen? A threshold alarm would need to be set so tight it fires on noise, or so loose it misses real events. The Kalman filter sidesteps this trade-off — it knows what it predicted, and the chi-squared test fires when reality disagrees.

This recipe uses a single Kalman 1d node with innovation gating. Toggle between follow mode (track the jump to the new level) and exclude mode (reject it as an outlier and continue at the old level) — same node, one parameter.

flow('shift-detector')
    .sanitize('sane', 'level', { failureReason: 'failReason' },
        { ranges: { level: { min: -50, max: 200 } } })
    .median3('m3', 'level', { median3: 'm3' })
    .kalman1d('kf', 'm3',
        { filtered: 'estimate', innovation: 'innovation',
          innovationGate: 'gateValue', variance: 'estVariance' },
        { followMode: true, sensorVariance: 4,
          processVariance: 0.01, chi2Threshold: 6.63 })
    .run()

Drag the slider and watch the filter track the signal — then see what happens at the outliers and the step change.

What You’re Seeing

The gray line is the raw level signal — stable around 50 with Gaussian noise, two isolated outlier spikes (samples 80 and 150), and an abrupt step change to 62 at sample 250.

The emerald line is the Kalman filtered estimate — it smoothly tracks the true level, rejecting noise. The indigo curve on the right axis is the innovation (prediction error) — flat near zero during normal operation, spiking when measurements violate the model’s expectation.

Rose verticals mark where the innovation gate exceeds the chi-squared threshold (6.63, corresponding to a 1% false alarm rate). The gate fires at both outlier spikes and at the step change.

Toggle the mode to see the difference:

• Follow mode: At the step change, the filter resets to the new level and tracks normally afterward. The gate fires once, then innovation drops back to zero.
• Exclude mode: The filter rejects the step as an outlier and continues estimating at the old level. Innovation stays elevated because every subsequent measurement disagrees with the model.

Where This Pattern Fits

How It Works

The Kalman filter maintains a state estimate (the filtered level) and an uncertainty estimate (the covariance). Each new measurement produces an innovation — the difference between what the filter predicted and what it observed.

The innovation gate normalises this surprise by the expected innovation variance (measurement noise + estimation uncertainty). The result is a chi-squared(1) statistic. At the default threshold of 6.63 (99th percentile), the gate fires when the innovation is larger than what the model can explain with 99% confidence.

In follow mode, a gate exceedance resets the filter to the new measurement — the filter accepts the jump and tracks from the new level. In exclude mode, the filter ignores the measurement and continues with its prediction — useful for rejecting sensor glitches while maintaining the true estimate.

The chi-squared threshold directly controls the false alarm rate: 3.84 = 5%, 6.63 = 1%, 10.84 = 0.1%. No tuning by trial and error — the statistics provide the guarantee (under Gaussian assumptions).

References

• Kalman, R.E. (1960). A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82(1), 35–45. doi:10.1115/1.3662552 
• Mehra, R.K. & Peschon, J. (1971). An innovations approach to fault detection and diagnosis in dynamic systems. Automatica, 7(5), 637–640. doi:10.1016/0005-1098(71)90028-8 
• Basseville, M. & Nikiforov, I.V. (1993). Detection of Abrupt Changes: Theory and Application. Prentice Hall. Free PDF 

Next Steps

• Detecting Gradual Drift — the complementary recipe for slow, invisible drift using fast/slow esMean crossover
• Detecting Sensor Freeze — detect when a sensor stops changing using running standard deviation collapse
• Under the Hood — understand what happens inside the pipeline