fix(stovepipe): mint a distinct message id for each buildsignal re-poll

[behinddwalls]

Summary

Why?

buildsignal schedules its next poll by re-publishing to its own topic, and it reused the build id as the message id — byte-identical to the message build published to start the loop. The MySQL queue dedups on the (topic, partition_key, id) unique key and InsertDelayed swallows the collision with ON DUPLICATE KEY UPDATE topic = topic, returning success. So the reschedule was accepted and silently discarded.

This is deterministic, not a race. The re-publish happens before the delivery is acked, and GC only collects up to the minimum acked offset on idle ticks, so the colliding row is always still present.

The effect: any build that is not terminal on its first poll is never polled again. Build.Status freezes at accepted/running, the request never leaves processing, nothing is published to record, and the queue's in_flight_count slot is never released — so after MaxConcurrent such builds the queue stops admitting work entirely.

Nothing caught it because the fake build runner could not report a non-terminal status until the previous commit, and the unit tests matched the published message with gomock.Any().

What?

publishBuildSignal now mints {buildID}/poll/{unixMilli}/{rand32}, following the precedent already established for the gate-wait reschedule in the process controller. The partition key stays the build id, so each build's poll loop keeps its own partition.

The timestamp is for human readability; the random suffix is what guarantees uniqueness. A bare millisecond stamp is not enough — a redelivery racing the original can publish twice within the same millisecond, which would resurrect the bug intermittently.

Test Plan

✅ bazel test //stovepipe/... — new unit test asserts successive re-polls mint distinct ids and never reuse the build id. Verified it genuinely regresses: reverting just the id line fails it with "map[bk-1:{}]" should have 3 item(s), but has 1.

✅ bazel test //test/e2e/stovepipe/... — new e2e ingests a queue carrying the build-slow marker and waits for the build row to reach succeeded, which requires more than one poll tick. The service log shows three distinct ids on the buildsignal topic for one build.

Stack

1. test(stovepipe): add build-slow marker to the fake build runner #464
2. @ fix(stovepipe): mint a distinct message id for each buildsignal re-poll #465
3. refactor(stovepipe): replace recorded greenness states with build outcomes #466
4. feat(stovepipe): record the build outcome on the request and free its slot #467
5. feat(stovepipe)!: key the record stage on the request id #468
6. fix(stovepipe): stop wrapping the buildsignal re-poll publish as retryable #469

[roychying]

in this case, we leave one build with two rows and two poll chains? letting dedup drop one seems fine? A build only needs one chain, and two just doubles the poll rate and races reconcile's CAS for no benefit.

my understanding is the potential value of the random suffix is the NTP case since time.Now() is wall clock..