feat: add adaptive context productivity evaluation - #99
Conversation
# Conflicts: # README.md # engraphis/core/store.py # engraphis/dashboard_assets/engraphis-graph.js # eval/agent_benchmarks.py # eval/benchmark.py # eval/context_economy.py # eval/longmemeval_v2_evidence.py # scripts/install_shortcuts.py # tests/test_benchmark_evidence.py # tests/test_context_economy.py # tests/test_eval_agent_benchmarks.py # tests/test_graph_engine_asset.py # tests/test_install_shortcuts.py # tests/test_longmemeval_v2_evidence.py # tests/test_receipts.py
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aa5a1c0f45
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for chunk in result.recall.chunks | ||
| if chunk.get("id") in packed_ids |
There was a problem hiding this comment.
When the context packer's utility ordering differs from recall rank—for example, when a lower-ranked but query-denser memory is packed first—the context labels that memory [1], but this comprehension still makes sources[0] the highest-ranked recalled chunk. Hosts resolving the ordinal against sources will therefore attach the wrong title, ID, and provenance; construct the list by iterating packed_chunks and joining chunk metadata by ID.
AGENTS.md reference: AGENTS.md:L152-L153
Useful? React with 👍 / 👎.
| if method == "full_history": | ||
| return history, "full_history", 0, "" |
There was a problem hiding this comment.
When a case's history exceeds max_context_tokens, the full_history branch returns the entire history unchanged; the correction path likewise passes the unbounded history directly at line 421. Those are precisely the workloads meant to exercise adaptive routing, but a hosted agent can reject the oversized prompt and the deterministic benchmark compares strategies under unequal context budgets, invalidating the resulting productivity numbers. Apply the configured cap, or explicitly reject over-budget tasks, on every raw-history attempt.
AGENTS.md reference: AGENTS.md:L154-L155
Useful? React with 👍 / 👎.
1 participant
Summary
Validation
python -m pytest tests/ -q --ignore=tests/test_hosted_evidence.py --ignore=tests/test_hosted_ledger.py --ignore=tests/test_hosted_luna.pypython -m pytest tests/test_adaptive_context.py tests/test_productivity_eval.py tests/test_receipts.py -qruff check engraphis/core/__init__.py engraphis/core/adaptive_context.py engraphis/core/engine.py engraphis/core/grounded.py engraphis/core/store.py engraphis/service.py eval/productivity.py tests/test_adaptive_context.py tests/test_productivity_eval.pypython -m eval.productivity --dataset eval/datasets/codemem.jsonl --max-context-tokens 512 --retrieval-token-budget 256Unrelated, untracked private-evaluation work remains intentionally excluded from this PR.