WP6 Digital Twins — What shipped this month

Sync LetsGrow into Red, the way Blue syncs Yookr

LetsGrow is Neurath's own climate computer, the system the grower uses to steer the greenhouse. It records climate, water, lighting and weather-station values. It also records the actuators and their setpoints: heating pipes, vents, screens, lamps and irrigation. Red's own sensors cannot see any of these. The SPoHF relay already serves this data as letsgrow-data. This PR passes it through the same client and sync orchestrator that Blue uses for yookr-data, and writes it into Red's database.

• What comes in: five LetsGrow modules (Climate B4, Grow light B4, Plant balance B4, Water B4, Weather station) with 51 declared sensors. Each module becomes a device (e.g. letsgrow-climate-b4), and each column becomes an lg_* sensor. The prefix keeps the names apart from Red's existing MySQL sensors, because Red routes queries by sensor name.
• Nothing silently dropped: a column that hasn't been declared yet is still stored (as lg_col_<id>) and logged.
• Local time: LetsGrow exports greenhouse wall-clock time rather than UTC, so timestamps are converted from Europe/Amsterdam time when the data is loaded. One limitation: on the night the clocks go back, the hour that occurs twice is stored only once.
• Daily delivery: data arrives once a day, up to local midnight. The Sync Status page therefore marks LetsGrow stale only after 36 h and an upstream outage after 60 h, instead of Yookr's 2 h/6 h.
• Running totals (radiation, PAR sum, irrigation, drain, water uptake) climb through the day and reset at midnight, so charts aggregate them with max, never average.
• Architecture: the sync orchestrator now groups each endpoint (feed) by twin and opens one database pool per twin. If one twin's database is down, only that twin's feeds fail. LetsGrow always writes to Red's database, so it cannot end up in Blue's.
• Deployment: a new hourly wp6-data-sync-letsgrow CronJob. Loading the history is a one-off full sync going back to at least 1 January 2024 (about 14 million rows).

Status: the CI check passed against its base (15 October). main has moved 7 commits since then, but the PR still merges without conflicts. It is waiting for review. Open question: where exactly Neurath's LetsGrow sensors hang has not been confirmed yet, so for now lg_par and lg_temp_greenhouse shouldn't be compared directly with Red's own s2100/s2101 sensors.

Wires come from upstream's sensor map

Three sensor wires are now installed in the greenhouse and behaving as they should. Getting there meant untangling a mix-up: from 13 July to 2 October, upstream's wire sensor map filed wire WS_01_03's climate sensors and its top PAR sensor under WS_01_01, which wasn't up at that time. Over that period Red showed three wires with gaps instead of two complete ones.

• Red now takes the list of installed wires from upstream's map (refreshed every 5 minutes) instead of a list kept by hand, which had drifted. A newly installed wire therefore shows up without a code change.
• The climate model no longer keeps its own list of which heights are available. It trains on every height of every installed wire, and the forecast shows exactly the heights that were fitted.
• A new coverage document records the mix-up and keeps a snapshot of the old map. That snapshot is the only way left to decode the misfiled July–October rows.

Fitted models stay in memory when the object store is down

When the educloud storage bucket refused connections, /dli returned an error on every request: the page went back to storage each time and failed. Startup could not recover either, and the nightly refit trained the climate model and then lost it.

• A model is held in memory as soon as it is trained. Saving it to storage is now a best-effort extra, because storage exists so the next process can skip a refit, not so this one can serve pages.
• If storage can't be reached at startup, the dashboard treats it as "nothing stored" and fits the models in memory.
• The nightly DLI refit now runs even when the climate refit fails.
• This also covers the multi-height predictions, which are the climate chain's third link.

A faster climate forecast that survives a storage outage

Every climate forecast used to download the trained model chain from storage again, which took about 600 ms per request, and a storage outage made the page fail. The model is now read once per process and kept. A retrain, the nightly refit and startup all update that copy directly. If storage is unreachable before any model has loaded, both climate pages say so instead of crashing.