WP6 Digital Twins — What shipped this update

Both twins evacuated onto the v2 cluster

The old Kubernetes cluster had become a cluster running nothing but this project, kept alive purely for us. Rather than maintain it, everything was moved to the university's shared v2 platform so the old one can be switched off entirely.

Because the two clusters use different storage technology, the data could not simply be copied at disk level — it had to be exported and re-imported. That was done and checked record by record:

• Blue: 5,340,000+ sensor readings, the full pre-aggregated history, and the yearly long_data manual measurements — the one dataset whose source files exist nowhere but the database.
• Red: full greenhouse readings history.
• The sync job resumed from exactly where the restore left off, so there is no gap and no duplication at the seam.

The public web addresses were then pointed at the new cluster after being validated against it first. The old cluster still holds a complete second copy and is a one-step rollback.

Blue finally got its own database

In production, Blue had been writing into the database server's default administrative database — an accident dating back to its first deploy that every other environment (local development, the setup scripts, the documentation) disagreed with. Blue now has a proper wp6_blue database of its own, created automatically, exactly as Red always had.

Because Blue is the second twin needing this, the "create a twin's database" step became one shared, reusable piece of configuration instead of a copy-paste — so a third twin is a few lines, not a new script.

Six infrastructure faults the move exposed — all fixed

The new cluster stores data on real network block storage rather than plain folders on a machine. That difference is invisible until it isn't: six separate assumptions baked into the old setup broke, each one found and fixed during the move.

• The database refused to start — the new storage arrives with a system folder already present, which the database reads as "this directory isn't empty, I won't initialise here". Now it initialises in a sub-folder — but only where explicitly switched on, so the old cluster's live 2.8 GB database was not touched.
• The startup deadlocked — the database-creation step ran after the app was declared healthy, but the app could never become healthy without its database. It now runs first.
• Deploys hung — the dashboards hold an exclusive lock on their upload/export storage, so starting a replacement before stopping the old one could never work on this storage type. They now stop then start.
• Uploads and exports could not be written — the new storage arrives owned by the system administrator, and the twins deliberately run as an unprivileged user. Ownership is now set correctly at mount time.
• One deploy wedged indefinitely — a setup job that finished in about a second was being cleaned up so fast that the deployment tool never saw it succeed, and waited forever for a job that no longer existed.
• The nightly CSV exports had silently failed for 23 days (see below).

A fallback that stays a fallback

During the changeover, new builds were published to both clusters' image registries — a fallback you cannot ship a fix to is a weak fallback, and feature work continued while the migration ran. Once the new cluster took over production, that second publish was removed on purpose: the old cluster now freezes at its last known-good build instead of quietly rolling forward onto untested code.

Redesigned, source-aware sync health

The Sync Status card was rebuilt to answer one question at a glance: is this source healthy, and if not, since when and how badly?

• Automatic sources (the datalake feeds) get a data-freshness budget and show Healthy · Stale · Outage · Failing. The important distinction: a sync that keeps succeeding while the data behind it stops advancing is now flagged as stale/outage — an upstream problem — rather than looking indistinguishable from a healthy source.
• Manual sources (uploaded spreadsheets, insect-trap counts) are event-driven, so they show upload recency instead of being wrongly judged against a sync schedule.
• Each source keeps a rolling 30-day run history, feeding a 7-day reliability figure ("X of the last Y runs succeeded") and a records-per-run sparkline.
• Failing-since and a consecutive-failure count replace the previous all-time totals, so an old, long-recovered incident no longer colours today's picture.
• Fixed: a recovered source kept displaying a fossilised old error. Success now clears the error state.

The whole thing is shared platform code, so all three twins got it at once.

End-to-end tracing across dashboards, sync and CLI jobs

Every web request and every scheduled job now records a trace: a timeline of what it did, including its database queries and its calls out to the external data APIs. A slow page or a stalled sync run can be opened up and read directly.

• The 15-minute sync job produces one trace per run, with a segment per data source — so an upstream API being slow is visibly distinct from our own database being slow.
• The nightly export and manual-ingest command-line jobs are traced too, and gained proper structured logging they previously lacked.
• Logs are stamped with their trace ID, so a log line links straight to the request that produced it.
• Health-check and metrics endpoints are excluded, and per-request internal noise dropped, so each request reads as one clean entry rather than three.
• Vendor-neutral by design: the twins emit the open OpenTelemetry standard to whatever collector is configured, and do nothing at all when none is. It switched itself on when the new cluster supplied an endpoint — no application configuration was needed.