Fix CAPI reasoning E2E fixtures

[ellismg]

Summary

The C# CAPI E2E fixtures configured explicit high reasoning effort with models that do not support configurable reasoning:

• StreamingFidelityE2ETests.Should_Emit_Streaming_Deltas_With_Reasoning_Effort_Configured inherited the fixture default, claude-sonnet-4.5.
• SessionE2ETests.Should_Set_Model_With_ReasoningEffort switched to gpt-4.1.

Both tests now use the reasoning-capable gpt-5.4 model. The streaming replay fixture also advertises gpt-5.4 so the recorded CAPI model catalog matches the test configuration. Assertions that verify high reasoning effort propagation remain unchanged.

This preserves the intended runtime contract introduced by github/copilot-agent-runtime commit 6c6125edb8 / PR #12016: explicit unsupported CAPI reasoning efforts fail instead of being silently discarded or clamped.

Validation

• dotnet test dotnet/test/GitHub.Copilot.SDK.Test.csproj --no-restore --filter 'FullyQualifiedName~StreamingFidelityE2ETests.Should_Emit_Streaming_Deltas_With_Reasoning_Effort_Configured|FullyQualifiedName~SessionE2ETests.Should_Set_Model_With_ReasoningEffort' (2 passed)
• dotnet format dotnet/test/GitHub.Copilot.SDK.Test.csproj --verify-no-changes --no-restore

[Copilot]

Pull request overview

Updates .NET E2E tests to use a reasoning-capable model.

Changes:

• Switches reasoning-effort tests to gpt-5.4.
• Aligns the streaming replay fixture’s model catalog.

Show a summary per file

File	Description
test/snapshots/streaming_fidelity/should_emit_streaming_deltas_with_reasoning_effort_configured.yaml	Advertises gpt-5.4.
dotnet/test/E2E/StreamingFidelityE2ETests.cs	Configures the streaming test with gpt-5.4.
dotnet/test/E2E/SessionE2ETests.cs	Uses and asserts gpt-5.4 during model switching.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Balanced

[stephentoub]

@copilot does the same issue exist in any of the other 5 language test suites?

[Copilot]

@copilot does the same issue exist in any of the other 5 language test suites?

Yes — the same issue existed in all five other SDKs. Fixed in commit 729182e.. (latest push):

setModel with reasoning_effort: "high" using gpt-4.1 (which doesn't support configurable reasoning):

• Node.js should set model with reasoningEffort → gpt-5.4
• Python test_should_set_model_with_reasoning_effort → gpt-5.4
• Go TestSetModelWithReasoningEffortE2E → gpt-5.4
• Rust should_set_model_with_reasoningeffort → gpt-5.4
• Java: no equivalent test exists

Streaming fidelity tests (no explicit model → implicit claude-sonnet-4.5 before the snapshot was updated):
All five SDKs now explicitly specify model: "gpt-5.4", matching the .NET pattern and making the reasoning-capability requirement self-documenting. The shared snapshot was already updated to advertise gpt-5.4 in the first commit, so these tests would have worked without the explicit model after that fix — but the explicit model is cleaner.

[Copilot]

Review details

• Files reviewed: 12/12 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (9)

dotnet/test/E2E/StreamingFidelityE2ETests.cs:158

• This still uses the class-shared runtime, whose 30-minute /models cache is normally primed by an earlier streaming fixture that advertises only claude-sonnet-4.5. Reconfiguring the proxy to this snapshot does not refresh that cache, so gpt-5.4 may be rejected or reverted and this E2E remains order-dependent. Use an isolated E2ETestContext here, as RpcSessionStateE2ETests.Should_Call_Session_Rpc_Model_SwitchTo does.

            Model = "gpt-5.4",

python/e2e/test_streaming_fidelity_e2e.py:166

• The module-scoped ctx.client retains the /models result from whichever earlier test starts it, and the earlier streaming snapshots advertise only claude-sonnet-4.5. Updating this test's proxy snapshot therefore does not make gpt-5.4 visible to that runtime, leaving this test order-dependent. Run this case in a fresh E2ETestContext, following the isolated model-switch test in test_rpc_session_state_e2e.py.

        session = await ctx.client.create_session(
            on_permission_request=PermissionHandler.approve_all,
            model="gpt-5.4",

rust/tests/e2e/session.rs:979

• The shared replay fixture test/snapshots/session/should_set_model_with_reasoningeffort.yaml still advertises only claude-sonnet-4.5. Model switching revalidates against that catalog and can emit this change event before reverting an unavailable target, so these assertions can pass without gpt-5.4 ever becoming active. Add gpt-5.4 to the fixture's models list so this actually exercises a reasoning-capable model.

                    "gpt-5.4",
                    Some(SetModelOptions::default().with_reasoning_effort("high")),
                )
                .await
                .expect("set model");
            let event = model_changed.await.expect("model change task");
            let data = event
                .typed_data::<SessionModelChangeData>()
                .expect("session.model_change data");
            assert_eq!(data.new_model, "gpt-5.4");

go/internal/e2e/session_e2e_test.go:1083

• The shared replay fixture test/snapshots/session/should_set_model_with_reasoningeffort.yaml still advertises only claude-sonnet-4.5. Model switching revalidates against that catalog and can emit this change event before reverting an unavailable target, so these assertions can pass without gpt-5.4 ever becoming active. Add gpt-5.4 to the fixture's models list so this actually exercises a reasoning-capable model.

	if err := session.SetModel(t.Context(), "gpt-5.4", &copilot.SetModelOptions{ReasoningEffort: copilot.String("high")}); err != nil {
		t.Fatalf("SetModel returned error: %v", err)
	}

	select {
	case evt := <-modelChanged:
		md, mdOk := evt.Data.(*copilot.SessionModelChangeData)
		if !mdOk || md.NewModel != "gpt-5.4" {
			t.Errorf("Expected newModel 'gpt-5.4', got %v", evt.Data)

go/internal/e2e/streaming_fidelity_e2e_test.go:294

• This subtest reuses the outer client, whose /models cache is already populated by an earlier streaming subtest with only claude-sonnet-4.5. Changing the replay snapshot does not refresh that runtime cache, so selecting gpt-5.4 is order-dependent. Give this subtest its own context/client, as the model-switch subtest in rpc_session_state_e2e_test.go does.

		session, err := client.CreateSession(t.Context(), &copilot.SessionConfig{
			OnPermissionRequest: copilot.PermissionHandler.ApproveAll,
			Model:               "gpt-5.4",

nodejs/test/e2e/streaming_fidelity.e2e.test.ts:153

• Creating another client wrapper is insufficient in the in-process E2E matrix: both clients load into the same host process and use the same token/proxy URL, so the runtime's process-wide /models cache retains the earlier claude-sonnet-4.5 catalog. This case remains transport-dependent even though its snapshot advertises gpt-5.4. Isolate it with its own createSdkTestContext() (and therefore a unique proxy/cache key), as the RPC model-switch test does.

        const reasoningClient = createClient();
        onTestFinished(() => reasoningClient.stop());
        const session = await reasoningClient.createSession({
            onPermissionRequest: approveAll,
            model: "gpt-5.4",

nodejs/test/e2e/session.e2e.test.ts:975

• The model-switch fixture still advertises only claude-sonnet-4.5, and this test also uses the describe-shared runtime whose /models cache may already come from another snapshot. session.model.switchTo can emit the change event and then revert a target absent from that cached catalog, so these assertions can pass without gpt-5.4 becoming active. Add gpt-5.4 to the shared snapshot and run this case in a dedicated createSdkTestContext(), following the isolated RPC model-switch test.

        await session.setModel("gpt-5.4", { reasoningEffort: "high" });

        const event = await modelChangePromise;
        expect(event.data.newModel).toBe("gpt-5.4");

python/e2e/test_session_e2e.py:701

• The model-switch fixture still advertises only claude-sonnet-4.5, and the module-scoped client may already have cached another snapshot's catalog. Model switching can emit this event and then revert a target absent from the cached catalog, so the test can pass without gpt-5.4 becoming active. Add gpt-5.4 to the snapshot and use a fresh E2ETestContext, following the isolated RPC model-switch test.

        await session.set_model("gpt-5.4", reasoning_effort="high")

        data = await asyncio.wait_for(model_change_event, timeout=30)
        assert data.new_model == "gpt-5.4"

dotnet/test/E2E/SessionE2ETests.cs:621

• The model-switch fixture still advertises only claude-sonnet-4.5, and the class-shared runtime may already have cached another snapshot's catalog. Model switching can emit this event and then revert a target absent from the cached catalog, so the test can pass without gpt-5.4 becoming active. Add gpt-5.4 to the snapshot and use an isolated E2ETestContext, following Should_Call_Session_Rpc_Model_SwitchTo.

        await session.SetModelAsync("gpt-5.4", "high");

        var modelChanged = await modelChangedTask;
        Assert.Equal("gpt-5.4", modelChanged.Data.NewModel);

• Files reviewed: 12/12 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (4)

go/internal/e2e/streaming_fidelity_e2e_test.go:294

• The parent test reuses one client across all subtests, so /models has already been cached from the first claude-sonnet-4.5 fixture before this snapshot is selected. The cache is not invalidated when ConfigureForTest changes snapshots, leaving gpt-5.4 unavailable. Give this subtest its own TestContext and client, following the isolated model-switch subtest in rpc_session_state_e2e_test.go.

			Model:               "gpt-5.4",

dotnet/test/E2E/StreamingFidelityE2ETests.cs:155

• E2ETestFixture.Client is persistent for this class, while the runtime caches /models for 30 minutes. Whichever test initializes the cache first makes either this gpt-5.4 fixture or the other Claude fixtures unavailable, so this class becomes order-dependent. Create an isolated E2ETestContext/client for this test, as RpcSessionStateE2ETests.Should_Call_Session_Rpc_Model_SwitchTo does.

            Model = "gpt-5.4",

python/e2e/test_streaming_fidelity_e2e.py:166

• This test still uses the module-scoped ctx/CLI, whose 30-minute /models cache was populated by an earlier streaming fixture with claude-sonnet-4.5. Reconfiguring the proxy to this snapshot does not refresh that cache, so gpt-5.4 is unavailable and the reasoning test is order-dependent. Run this case in an isolated E2ETestContext, as the model-switch test does.

            model="gpt-5.4",

java/src/test/java/com/github/copilot/StreamingFidelityTest.java:257

• The class shares one proxy URL across tests, but /models is cached for 30 minutes by (auth, base URL). Switching this snapshot to only gpt-5.4 therefore makes the result depend on which test populates the cache first (and is especially problematic for the in-process backend). Use a dedicated E2ETestContext/proxy for this test rather than the class-level ctx.

                            .setModel("gpt-5.4").setStreaming(true).setReasoningEffort("high"))

• Files reviewed: 12/12 changed files
• Comments generated: 5
• Review effort level: Balanced

[github-actions]

Cross-SDK Consistency Review ✅

This PR updates E2E test fixtures symmetrically across all six SDKs:

SDK	Model switch	Isolation refactor
Node.js	gpt-4.1 → gpt-5.4 ✓	Isolated describe block ✓
Python	gpt-4.1 → gpt-5.4 ✓	Isolated E2ETestContext ✓
Go	gpt-4.1 → gpt-5.4 ✓	ConfigureForTest + sub-test ✓
.NET	gpt-4.1 → gpt-5.4 ✓	Isolated E2ETestContext ✓
Java	gpt-4.1 → gpt-5.4 ✓	(streaming fidelity only) ✓
Rust	gpt-4.1 → gpt-5.4 ✓	(no change needed) ✓

Both shared YAML snapshots (session/should_set_model_with_reasoningeffort.yaml and streaming_fidelity/should_emit_streaming_deltas_with_reasoning_effort_configured.yaml) are updated consistently.

No cross-SDK consistency issues found. The changes are well-aligned across all language implementations.

Generated by SDK Consistency Review Agent for #2181 · sonnet46 18.1 AIC · ⌖ 5.42 AIC · ⊞ 6.6K · ◷

[Copilot]

Review details

Suppressed comments (3)

python/e2e/test_streaming_fidelity_e2e.py:166

• This still uses the module-scoped ctx.client, whose 30-minute /models cache is warmed by earlier streaming tests from snapshots that advertise only claude-sonnet-4.5. Reconfiguring the same proxy to this GPT-only snapshot does not change that cache key, so creating this session with explicit high reasoning can fail as unsupported in a full module run. Give this test its own E2ETestContext, as the model-switch test in test_session_e2e.py now does.

            model="gpt-5.4",

go/internal/e2e/streaming_fidelity_e2e_test.go:294

• The enclosing test reuses one client across all subtests, so earlier Claude-only streaming snapshots populate the runtime's /models cache for this proxy URL. This GPT-only snapshot cannot refresh that cached catalog, and explicit high reasoning may therefore be rejected. Create a fresh context/client inside this subtest so it has a distinct proxy/cache key.

			Model:               "gpt-5.4",

dotnet/test/E2E/StreamingFidelityE2ETests.cs:158

• CreateSessionAsync uses this class fixture's persistent client and proxy, whose /models cache may already have been populated by another streaming test's Claude-only snapshot. Switching the proxy fixture here does not change the cache key, so the full class run can reject high for gpt-5.4. Use an isolated context/client for this test, matching Should_Set_Model_With_ReasoningEffort.

            Model = "gpt-5.4",

• Files reviewed: 13/13 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

• Files reviewed: 13/13 changed files
• Comments generated: 0 new
• Review effort level: Balanced