Recipe · subsampling

Trajectory-aware adaptive compression

The deadband recipe is a good first cut, but it has a blind spot: every decision is local. It does not know which way the signal is heading, how confident the model is, or whether a sample sits at an inflection. The cost shows up at step changes — the deadband widens just after the event, and the reconstruction trails behind. The fix is to give the flow a model.

This recipe wires seven nodes into a trajectory-aware subsampling flow. A kalman1d predictor with followMode runs alongside esStats and trend; the winnow node fuses the slope, the noise estimate, and the innovation gate into a single per-sample keep decision — its one-sample buffer anchors each spike to its baseline neighbor; passIf forwards the kept samples; a second kalman1d smooths measurement noise from the kept values before storage.

flow('adaptive-compression')
    .median3('med', 'value', { median3: 'smoothed' })
    .kalman1d('kf', 'value',
        { filtered: 'filtered', innovation: 'innovation',
          innovationGate: 'gate' },
        { sensorVariance: 0.005, processVariance: 2e-5,
          chi2Threshold: 6.63, followMode: true })
    .esStats('stats', 'smoothed',
        { stdev: 'stdev', mean: 'mean' },
        { halfLife: 50 })
    .trend('slope', 'smoothed',
        { trend: 'trendDir', confidence: 'trendConf', rocMean: 'roc' },
        { rocStatsHalfLife: 20, rocThreshold: 0.005, warmupSamples: 15 })
    .winnow('compress', 'smoothed',
        { significant: 'store', deviation: 'dev', predicted: 'pred',
          xPrev: 'prevValue', tPrev: 'prevTimestamp' },
        { K: 2, tightenBase: 100, maxGap: 50, chi2Threshold: 12,
          bufferPrev: true, timestampField: 'timestamp' })
    .passIf('gate', (msg, counter) => msg.store === true)
    .kalman1d('kfSmooth', 'value',
        { filtered: 'storedValue' },
        { sensorVariance: 0.005, processVariance: 0.0025,
          chi2Threshold: 6.63, followMode: true })
    .run()

On the deadband’s signal

The deadband recipe runs on a four-region test signal — quiet baseline, three pulses, a slow ramp, a noisier regime. Drag the slider to the end and compare the KPI numbers below with those on the deadband page.

On a harder signal

Now watch what happens on a signal with two sharp step changes — one up at sample 200, one down at sample 350 — plus high-amplitude vibration between them. Drag the slider through all five regions.

What You’re Seeing

Both charts show the same three elements: the cyan line is the original signal, the orange line is the linear-interpolation reconstruction between consecutive kept samples, and the amber dots mark the samples the flow decided to store.

First chart (deadband’s signal). The same quiet-baseline, pulses, ramp, and noisy-regime signal the deadband recipe uses. Compare the KPI numbers directly: the adaptive flow achieves tighter reconstruction error because winnow fuses trajectory, noise estimate, and innovation into each keep decision — not just deviation from a running mean.

Second chart (harder signal). A quiet sinusoid, a slow ramp, a sharp step UP at sample 200, high-amplitude vibration, then a sharp step DOWN at sample 350. Two structural breaks in opposite directions — this is where the trajectory awareness earns its keep. At each step, the Kalman innovation gate trips; KF1’s followMode snaps the predictor to the new level in one tick; winnow forces a kept sample so the reconstruction tracks the step instead of trailing through it. A local-only gate like the deadband would lag behind both transitions — the running mean cannot keep up with instantaneous structural breaks.

The KPI strip under each chart shows three numbers: the compression ratio, the largest single-sample reconstruction deviation, and the RMS reconstruction error in the same units as the signal.

Where This Pattern Fits

How It Works

The seven nodes split the work into single-purpose pieces. median3 absorbs single-sample spikes. The first kalman1d predicts the signal’s trajectory from a constant-velocity model and fires its innovation gate (a chi-squared test) when reality disagrees with the prediction — that gate is what catches step changes the deadband alone would miss. KF1’s followMode snaps the filter to the new measurement when the gate fires. The predictor tracks the jump in a single tick instead of staying anchored to the old level.

esStats maintains the running mean and stdev that set the local noise scale. trend classifies the signal’s direction and its rate of change. winnow is the decision node — it fuses the slope, the noise estimate, the trend direction, and the innovation gate into a single per-sample store flag. Its one-sample buffer (bufferPrev) publishes the previous tick’s value on gate-fire keeps, giving the reconstruction an anchor at the sample before each spike. passIf drops every sample whose store flag is false. The second kalman1d reads from the raw value (not the median3 output) and smooths measurement noise from the kept samples before storage.

The single sensitivity parameter is K on winnow. K = 2 means “store when the deviation exceeds twice the current noise floor.” Because the comparison is multiplicative on the running stdev, the threshold scales with whatever the local noise happens to be — there is no absolute tolerance to pick per signal. K itself is dataset-specific: the value here is tuned on the NASA IMS bearing data, and other datasets should sweep K on their own representative signals before deploying.

The first kalman1d reads from the raw value, not from the median-smoothed stream. The Kalman filter’s sensorVariance parameter is the right place to model measurement noise — pre-filtering with median3 would double-filter and leave the chi-squared innovation gate (6.63, a 1% false-alarm test) under-calibrated. esStats and trend still read from smoothed so their running statistics stay noise-protected.

KF1 and winnow each have their own chi2Threshold. KF1’s threshold (6.63) controls detection sensitivity — how surprising a sample must be before the innovation gate fires and followMode snaps the predictor. winnow’s threshold (12) controls storage sensitivity — how extreme a gate fire must be before it forces a kept sample. On vibration-only signals, set both to 6.63. On signals with step changes and high post-step noise, raise winnow’s threshold so it stores structural breaks without keeping every noisy post-step sample.

winnow’s maxGap = 50 guarantees at most 50 samples between kept points — a minimum sampling-rate fallback that rarely fires on vibrating data but prevents the reconstruction from flat-lining through a long quiet stretch.

Like every recipe, these parameter values are a starting point. Adapt K, the chi-squared thresholds, and the Kalman variances to your own signal characteristics and storage budget before deploying.

References

• Bristol, E.H. (1990). Swinging Door Trending: Adaptive Trend Recording? ISA National Conference Proceedings, pp. 749–756.
• Kalman, R.E. (1960). A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering, 82(1), 35–45. doi:10.1115/1.3662552 
• Welford, B.P. (1962). Note on a method for calculating corrected sums of squares and products. Technometrics, 4(3), 419–420. doi:10.1080/00401706.1962.10490022 

Next Steps

• Subsampling Noisy Signals with a Deadband — the four-node entry point that this recipe builds on
• Frozen Sensors — the inverse problem, when a signal has stopped changing
• Composition Patterns — understand how independent nodes compose into richer flows