Correct the fetch cache's memory bound (#364)

[ChronicallyJD]

What

Closes #364 by correcting the claim rather than changing the behaviour, which is what
you asked for on #367 ("I would like the fix to correct that specific sentence rather
than only the code").

#361 states in its commit message and in design/ISSUE_359_FETCH_CACHE_PARTIAL.md
that the per-column release keeps the cache bounded by 4 x the cap. It does not.

Why the bound does not hold

The release trims the decoded streams. It cannot trim rankPrefix and valOffset,
which stay in the entry context deliberately so a released column keeps constant-time
row reach — that part of the design is right and this PR does not touch it.

But valOffset is four bytes per value per varlena column, ~600 KB per column at
the default stripe_row_limit. Enough varlena columns and the retained indexes alone
exceed the cap, after which every newly decoded column is released immediately and the
entry stops shrinking.

Measured, 150,000 rows x 60 text columns, forced index scan over 200 rows:

columnar fetch column | n=2 | total=18 MB     <- the columns that stayed resident
columnar fetch group  | n=1 | total=44 MB     <- groupBuffer + retained indexes
ALL columnar fetch contexts total: 62 MB      <- against a 32 MB cap

Real bound: 4 x (cap + retained position indexes + groupBuffer). On that shape the
speedup is also only 1.14x (127,699 ms against 145,132 ms before #361), because
almost everything overflows.

Why not fix the bound instead

I built that and measured it, and it is the wrong trade:

build	memory held	time
before #361	nothing retained	145,132 ms
#361 as merged	62 MB vs a 32 MB cap	127,699 ms
releasing valOffset too	28 MB, under the cap	187,521 ms

47% slower than #361 and 29% slower than the behaviour #361 replaced. My reasoning
that rebuilding offsets was "a constant factor on a decode already happening" was
wrong — it is a second O(values) walk plus a 600 KB allocation, per overflowed column,
per fetch. The retained indexes are precisely what makes an overflowed column cheap on
its next fetch.

Holding offsets for 60 varlena columns at a 150,000-row group needs ~34 MB and does
not fit in a 32 MB cap. The bound and the speed are in real tension; #361 chooses the
speed, and this records that choice instead of asserting a bound that does not hold.

If you would rather the cap be a real ceiling, option (2) on the issue — drop the
entry on its actual footprint rather than on raw bytes — returns this shape to ~145 s
and I will build it. That is a design call and it is yours.

Scope

Comment and design-doc only. No behaviour change, so no new test: there is nothing
new to assert, and the numbers above are reproducible from the fixture described in
the doc. The commit message of #361 is immutable, so the correction lives where a
reader will actually hit it — the code comment at the drop site and the design doc's
memory-bound section.

🤖 Generated with Claude Code

[jdatcmd]

Approved. My defect, correctly diagnosed, and the honest resolution.

Zero executable lines change — the diff is the design note plus one comment
block, and the rg->byteLength > COLUMNAR_FETCH_CACHE_MAX_BYTES guard is untouched.
So this carries no behavioural risk, and CI green is sufficient verification.

The claim it corrects is mine. #361's commit message and design note both said the
cache "keeps the bound at 4 x the cap", and that sentence is false for exactly the
reason you found: the per-column release trims decoded streams, and I deliberately
kept rankPrefix/valOffset in the entry context so a released column keeps
constant-time row reach. I reasoned about the streams and did not carry the
retained indexes into the bound. valOffset at four bytes per value per varlena
column is not a rounding error at 60 columns.

Three things I want to note, because they are why this is the right patch rather
than a note-to-self:

• You measured the alternative instead of asserting it. Releasing valOffset
  with the stream holds the bound (62 MB to 28 MB) and costs 47% in time (127.7 s
  to 187.5 s). That is the fix I would have reached for first, and it is worse. The
  comment now says so with numbers, so the next person does not re-derive it.
• The real bound is stated as a formula, 4 x (cap + retained indexes + groupBuffer), rather than as "it is unbounded" or a hand-wave. That is a bound
  someone can design against.
• You left the trade explicit rather than resolving it silently. "This design
  chooses the speed" with the tension named is the correct outcome when the two
  goals genuinely conflict at 60 varlena columns.

The 1.14x on that shape is worth keeping visible too. #361 is a large win on the
narrow-projection case it was written for and nearly nothing on this one, and both
being in the record is the difference between a fix and a claim.

Merging.