main · 34 commits · 0 merged PRs · 1 contributor🔗 The Open live buttons (↗) deep-link into the deployed dashboards. Blue and Red require a login on first open; Grey is a public demo (no login). Model pages are admin-only.
It began as one odd report: on long summer days the lamps are off, yet the forecast page showed three different numbers where two should match. Following that up turned up a series of quiet errors in how light was measured and scored.
Red has two light sensors, one at the roof above the lamps and one at plant level below them. Until now the lamps' share of the light was worked out as "whatever the model got wrong". A model error is not a lamp, and this method could never report "lamps off": on a summer day with a perfect daily prediction it still made up a 5.5 mol lamp contribution.
The new rule is simple and physical. If the plants get more light than the roof sensor above them, a lamp must be on, because glass and screens can only take light away. Judged over a rolling two-week window per hour of the day, it finds:
The forecast was also being given different weather inputs than it was trained on: total sunlight in place of direct sunlight, and a fixed 50% cloud cover. It now gets the real inputs, and refuses to predict if one is missing, rather than quietly filling in a value that looks plausible.
af710f8 fix(red): the light model — lamps measured, not inferred from model error94c64e3 feat(shared): resample readings to any interval, and encode hour of dayOne square for every hour of every day, coloured by what lit the crop: lamps only, daylight + lamps, daylight or dark. You can check the lamp detection by looking at it instead of taking it on trust. On December data it shows lamps from 23:00 to 06:00, daylight plus lamps until 15:00 and darkness after that. Toward February the 16:00 hour switches on as the days get longer, which nothing was programmed to show.
Days when the sensors weren't measuring (they were outside the greenhouse from 27 May to 12 July) are left out instead of being drawn as dark nights.
61ed817 feat(red): a lamp calendar — which hours the lamps actually rana2d42ab feat(shared): let a page say where its "All" date preset beginsThe DLI model used to report only how well it fitted its own training data. It is now tested the way the climate model is: on whole days held out from training, and compared with the two simple forecasts any model has to beat, "same as yesterday" and "the usual for this time of year". The first honest result was uncomfortable. The weather step had no real skill (it had memorised its training data), and the full chain only matched "same as yesterday".
The main cause turned out to be the data, not the model. A daily total was computed by adding up sensor readings, so it partly measured how often the sensor reported. In June the roof sensor dropped from ~270 to 97 readings a day, and the sunniest days of the year came out at a third of their real value. Daily totals are now proper integrals over time:
The performance page had three related fixes. It compared a plant-level prediction with the roof sensor, which looked like a 38% underprediction but was a units mismatch. It counted sensor outages as perfect days. And it drew outages as days of zero light. Outages now show as a labelled gap in the line and are left out of the score.
e724186 feat(red): DLI chain beats persistence — a daily total is an integral5c6bc56 feat(red): DLI scored out of sample — and stage 1 has no skill at alla977724 fix(red): DLI performance — compare like with like, and stop scoring outagesda6cdcd fix(red): an outage is a gap in the line, not a day of zero lightThe two light lines were labelled "Natural Light" and "Total Light", which suggests the second includes the first. Whenever the sun was out, the plant-level line fell below the roof line and the chart looked self-contradictory. They are now Light above lamps and Light below lamps, at plant level, with a short explanation of why the crop gets less light than the roof.
The "Lamp DLI" column, which had been calculated as total minus natural, went negative all summer and was too high in winter. It now uses the measured lamp contribution described above.
49ac8b1 fix(red): DLI history — label the sensors by height, measure the lampThe goal is to predict the climate at the level of the plants. These features move from greenhouse-wide averages to what each height and each fruit cohort actually experienced.
A three-step model predicts temperature, humidity, CO₂ and light at each height on the measuring wires:
The model is split where the data is. The wires have about nine usable summer weeks and the greenhouse sensors have eleven months, so only the smallest step learns from the wires. Light reuses the measured lamp contribution from the DLI work instead of trying to learn the operators' lamp schedule from the weather. Every score is tested on held-out whole days against simple baselines.
Growers see a band chart per height on /climate/forecast. Admins see the skill grids and retraining suggestions on /climate/model/. The full method is written up in the model paper in the repository.
6ca623e feat(red): climate model — weather to greenhouse level to per-height wireee0d079 refactor(red): one forecast chart, not two — and mend the outdoor lineThree measuring wires hang in the greenhouse, but every multi-height page made you pick one, so you couldn't ask whether the greenhouse is uniform. This page shows all three side by side, per growth section. Each wire gets a verdict that separates a constant offset (how it hangs or is calibrated) from a divergence at one height (a genuinely different microclimate there).
Colours use a fixed scale, so a calm day and a drifting day can't look alike. Every number shows how much data is behind it. Against production data it immediately showed two things:
fce36c7 feat(red): uniformity across wires — where the greenhouse agrees, and where it doesn'tA tomato plant sets a new truss every week, and each one ripens over about eight weeks, so several "cohorts" are always growing at once. The crop-cycles page gives each cohort its own lane, coloured by the climate it lived through. Its measured outcome (Brix, weight, …) is shown at the harvest end, so you can see which cohorts were exposed to a particular spell of weather.
4625820 feat(red): crop cycles waterfall, on a shared cycles/cohorts model45aae7c fix(red): one marker per cohort and cultivar, aggregated as the measure declares761fa28 fix(shared): a value chip is a label, not a lidbe4c565 feat(red): say where the climate on crop-cycles was measured8db3f65 / 7a30286 / 315948c blue seasons page: built, fixed, revertedThe final 2025/26 fruit-quality dataset added a converted-°Brix column, so production rejected the upload. Brix is now ingested (all 128 rows land between 2.6 and 6.2 °Brix, the normal range for tomato). The rejection message is fixed too: it used to report the last column as missing, which pointed away from the real cause. It now names the added column by its Excel letter.
a01aefc feat(red): ingest Sijia's Brix column, and say which column driftedUntil now, every deploy was a short outage, and every restart threw away the trained models. Both have been fixed.
Each dashboard kept its files (nightly exports, trained models, uploaded spreadsheets) on a disk that can only attach to one machine at a time. So the old version had to stop before the new one could start, and every deploy meant downtime. It is also why the nightly exports silently failed for 23 days after the cluster move (see the previous update).
In three steps, all of those files moved to the university's object storage:
Both dashboards now attach no disks at all. A deploy starts the new version, waits until it is ready, and only then stops the old one. The first rolling deploy served every request to Red but dropped one request on Blue, because the old pod shut down a moment before the router noticed. A short pause on shutdown closed that gap.
519e24f feat: CSV exports live in the bucket, so the export job stops being pinned to a node0e455dd feat(red): trained models live in the bucket, and red is down to one PVCc68af06 feat(blue): soil models in the bucket — both twins now hold one PVC eachb6d8e66 feat: uploads in the bucket — no PVCs left, and the dashboards finally roll0de5c75 fix(helm): preStop sleep, so a rollout drops nothing at all9a543d9 fix(ci): swap MinIO for adobe/s3mock — MinIO's images need auth nowUploads are stored by content, so uploading the same file twice gives two records that point at one stored file. Blue's yearly long_data workbook is re-uploaded like that on purpose. The clean-up of old uploads deleted the file without checking whether a newer record still needed it, which is how Blue's 2024 workbook and Red's original fruit-quality file had silently disappeared. Clean-up now keeps any file that a kept record still uses. The missing files were restored from operator copies and matched against their checksums, so every live upload record has its file again.
b6d8e66 feat: uploads in the bucket — no PVCs left, and the dashboards finally rollModels used to be lost on every deploy or restart and retrained on startup. For Red's climate model that takes ~17 minutes in the cluster, and during that time the page was "up" with nothing to show. Models are now kept, and retrained every night (Red 03:00, Blue 03:30 UTC) so they keep learning as new data comes in.
Keeping models brings a risk: serving a model trained for a different question. So each model stores a fingerprint of the settings it was trained with. If the configuration changes (a wider training window, a new forecast horizon, a different sensor), the model is retrained instead of serving stale answers. Earlier, Red's DLI model filled gaps in old files with guessed defaults, and once loaded a guess looks exactly like a fitted value. Old or mismatched models are now refused.
1e70f07 feat: trained models survive a deploy, and say what fitted themNow that exports, models and uploads all depend on object storage, its security certificate matters. The one in use at the time of writing expires on 11 October. The risky case isn't the expiry date but a renewal signed by an authority our software doesn't trust (one of the two roots involved isn't in the default list). So the check does a real connection test the same way the twins connect, and shows both the days left and who signed the certificate.
The Status page shows OK above 21 days, RENEW SOON below that and ACTION NEEDED under 7 days or if the connection test fails. The check can never crash the Status page itself.
1aada56 feat: watch the object store's certificate, which expires in 11 daysThe first look at yield per plant, and support for this year's pooled fruit-quality samples.
A new Manual Monitor page shows, for each year, how per-plant season yield varies within each treatment, and how each pick date adds to the season total. A plant's season yield is the sum of its picks, so the plant number from the long_data workbook is now kept as a sample number. Treatments are listed in field-layout order with their plot codes. The Manual Monitor now takes its years from the data, so it no longer stops at 2025.
cb5ca1d feat(blue): harvest per treatment, per plant and per pick dateIn 2026 the berries of three treatments are pooled before measuring weight, Brix and firmness: Mix_D (Ca + V_CA + V_CA_G_BrPK) and Mix_E (K + G_K + V_K_G_CaBrP). These are now separate pooled groups, because a blend can't be credited to any one of its members. Before this, 180 rows of the 2026 file were skipped as unknown.
b0689bc feat(blue): accept 2026's pooled Mix_D/Mix_E fruit-quality samplesLess visible, but it keeps the twins trustworthy and the codebase maintainable.
94c64e3 feat(shared): resample readings to any interval, and encode hour of daya2d42ab feat(shared): let a page say where its "All" date preset begins439630d test(e2e): disarm blue's cagg refresh policy so it cannot race the tests893ed4f chore(scripts): add a red MySQL backup script57b22bc chore(scripts): check whether datalake offset pagination is safe yet00ad1c6 chore: drop the dead export-metadata helpers