Fix flaky sse-retry conformance test caused by CI timing sensitivity#1279
Open
Fix flaky sse-retry conformance test caused by CI timing sensitivity#1279
Conversation
The conformance test runner exits with code 1 when there are warnings, even if all checks pass. Timing-sensitive checks like SSE retry can produce warnings in CI due to network/processing overhead, causing flaky test failures. Parse the conformance output to detect warning-only results (0 failures, >0 warnings) and treat them as passing. Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Investigate and fix flaky tests in CI
Fix flaky sse-retry conformance test caused by CI timing sensitivity
Feb 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
sse-retryclient conformance test fails intermittently in CI because the conformance runner (v0.1.13) exits with code 1 for warnings, and network overhead on CI runners causes the SSE reconnection to arrive ~259ms late (759ms vs 500ms retry interval). The runner itself marks this as a WARNING ("acceptable but may indicate network delays"), not a FAILURE.0 failed, N warnings(N > 0) as passing, since warnings represent acceptable behavior per the conformance specReal failures (client doesn't reconnect, wrong behavior) still fail the test. Only timing-sensitive warnings caused by unavoidable network latency are tolerated.
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.