WP6 Digital Twins — What shipped this update

Soil-condition forecast (next 7 days)

A new forecast page predicts soil moisture and soil temperature for day +1 through +7, one curve per treatment group. It learns from the last 30 days of readings (using lag features) and plots a shaded uncertainty band so you can see how confident each prediction is.

• Models retrain automatically on every deploy — no manual training step, and nothing to keep in sync on disk.
• An admin "Update" button forces a retrain on demand; concurrent clicks are safely serialised so two updates never clobber the same model files.

GDD growth tracker: full-year coverage from modeled weather

Growing Degree Days (a running measure of accumulated heat that drives crop development) used to read the farm's own weather-station sensor, which lacked complete coverage for 2024 — leaving gaps in the analysis. It now sources daily min/max temperature from OpenMeteo modeled weather at the farm's exact location (Grubbenvorst), stitching together the historical archive, the recent past, and a 14-day forecast into an unbroken series from 1 Jan 2024 onward. That gives Blue a full calendar year of GDD to analyse for the first time.

• Growth constants set to recognised values — a 5 °C base temperature (European convention, biofix 1 Jan) and a 7.2 °C chill-hour threshold.
• GDD left the data-source toggle entirely and now has its own card on the Blue home page.

Fertigation events overlaid on soil charts

Irrigation/fertigation events (uploaded as CSV or via the admin UI) now appear as vertical markers on the soil monitor, so a moisture spike lines up visually with the watering that caused it. The overlay reads from the same ingested data store as every other manual dataset — one source of truth, not a separate CSV re-read.

Manual-measurement monitor & mixed views

A dedicated hub for the hand-recorded lab measurements, with correlation and comparison views — plus mixed views that put sensor data and manual data (or manual measurements and fertigation) on the same chart for direct comparison.

Broken / unreliable sensors tab

A view that surfaces sensors whose data is unreliable rather than simply missing — separating soil and weather feeds — so data-quality issues are visible instead of silently skewing charts.

Blue runs entirely on the datalake — Yookr fallback removed

The SPoHF Datalake is now Blue's single, canonical source of automated readings. The direct Yookr API — kept as a fallback while the relay caught up — is no longer needed and has been removed, along with the source toggle and the readings.project column that distinguished the two feeds.

• Code — 607 fewer lines of production code (289 added / 896 removed, measured across the retirement commits). Seven files deleted outright (619 lines of dedicated yookr code): the entire src/wp6_data/yookr/ client package, the 299-line sync orchestrator, blue's datasource.py + yookr.py, and the full-sync script. The remainder is net simplification of deps.py (every project filter gone), provider.py, ops.py and __main__.py — partly offset by the truncation fix below.
• Storage — 4,104,138 rows purged, exactly 43.5% of the readings table (9,432,610 → 5,328,472). The project column is dropped from every remaining row and the dedup index loses a column. daily_coverage shed 4,323 stale device/sensor/day entries (44,930 → 40,607) that no longer had readings behind them. The hypertable now stands at 2.77 GB, already vacuumed.
• Compute — one of two API syncs eliminated on every run. The sync job runs every 15 min (96×/day) and no longer authenticates against and paginates the direct Yookr API. A post-migration run ingested 27,078 records in 22.5 s. Two secrets (yookr-email, yookr-password) and one external dependency (api.yookr.org) are gone from the platform entirely.
• Simplification — the toggle machinery is gone. Two parallel data-source mechanisms collapsed to zero, two BlueSensorProvider instances to one, and a cookie-dispatched provider to a static badge. Blue now matches Red's single-categorical readings.source model, closing issues 024, 026 and 029 and rendering 027 moot.

277,588 sensor readings recovered — we were truncating our own ingest

Auditing the two feeds before the purge turned up a 337,081-reading gap. Most of it was not the relay's fault — it was ours. The SPoHF datalake is Elasticsearch-backed and refuses to page past 10,000 records in one query. Our client paged by offset until it got a short page, so any day holding more than 10,000 readings simply stopped early, with no error. We read "the server won't give me more" as "there is no more".

• Scale of the loss — 83% of days were affected. Two test-rig sensors (931k rows) ran from 2024-11-15 to 2025-12-31 and pushed daily volume over the cap for exactly that window. Re-syncing it with the fixed client split 330 of 396 days. Yookr-only readings fell from 337,081 to 59,493.
• The fix — bisect, don't truncate. fetch_window now detects that the next page would cross the cap and splits the time range in half, recursing until each half fits. It is self-correcting at any future data volume, rather than needing an operator to notice and re-tune a window size.
• Why nobody saw it. The coverage grid on /status matches at (device, sensor, day) granularity — a day counts as covered if it has at least one reading. Losing the tail of every day is invisible there. Only a row-level comparison against the second feed exposed it, which is the one thing retiring that feed would have made impossible. It was found with a week to spare.
• What could not be saved. December 2025 remains incomplete: the relay's stored history for that month still returns one sensor per device, so 58,826 readings across 36 series exist only in the retired feed. A backfill has been requested from SPoHF; the purged rows are archived.

Finishing the job — 255 lines of dead scaffolding removed

With one automated source per twin, a whole layer of the platform existed only to support switching between sources that no longer exist. None of it was broken — it was vestigial, the kind of code that misleads the next reader into thinking a feature is still there. A follow-up sweep removed it: 255 fewer lines of production code (126 added / 381 removed) with no loss of function.

• The source toggle is gone for good. The cookie-based switcher, its client-side JS, the per-source provider dispatch, and a data-source label threaded through ~40 render sites were all removed. Each dashboard keeps a simple static badge naming its source; nothing a user sees changed.
• One coverage store instead of two. A `daily_coverage` table duplicated exactly what the daily-summary aggregate already held (verified identical on prod, 40,607 rows either way), and — being append-only — it could keep claiming days whose data had been deleted. Coverage now reads from the aggregate alone, which stays honest as data changes. That retired a table, two write paths, a rebuild job, and an admin button.
• A dormant optimisation now works. The incremental sync's "stop early when a window is all duplicates" had never once fired, because of a Postgres quirk in how inserts-vs-updates were counted. Fixed — a steady sync now recognises 8,679-of-8,679 rows as already-present and can stop early.
• Both twins converge. Blue and Red now share one readings shape keyed on a single `source` column, so the platform's shared code no longer bends around a difference that used to exist only in Blue.