Skip to content

fix: Return HTTP response immediately without waiting for fault recovery - #565

Merged
yimingsu01 merged 13 commits into
mainfrom
claude/issue-244-20260213-2244
Feb 20, 2026
Merged

yimingsu01 merged 13 commits into
mainfrom
claude/issue-244-20260213-2244

Conversation

@HacksonClark

Copy link
Copy Markdown
Member

This change prevents the submission API from blocking while performing slow operations like fault injection/recovery and app teardown. These operations can take minutes when running multiple problems concurrently.

Changes

  • Add async cleanup method that performs fault recovery, undeploy, and cluster reconciliation in the background
  • Update _finish_problem to set state to "tearing_down" immediately and schedule async cleanup
  • Modify submit() to capture results snapshot and return immediately
  • Prevent noise restart during teardown state
  • Add "tearing_down" to possible stages in API documentation

Fixes #244


Generated with Claude Code

This change prevents the submission API from blocking while performing slow operations like fault injection/recovery and app teardown. These operations can take minutes when running multiple problems concurrently.

Changes:

- Add async cleanup method that performs fault recovery, undeploy, and cluster reconciliation in the background

- Update _finish_problem to set state to tearing_down immediately and schedule async cleanup

- Modify submit() to capture results snapshot and return immediately

- Prevent noise restart during teardown state

- Add tearing_down to possible stages in API documentation

Fixes #244

Co-authored-by: Jackson Clark <[email protected]>
@Saadmrp1038

Saadmrp1038 commented Feb 13, 2026 •

Copy link
Copy Markdown
Collaborator

Found an error while working.

2026-02-14 05:14:46,935 - clients.stratus.tools.submit_tool - INFO - HTTP submission failed: {'status': 'ok', 'text': 'Submission received'}

The submit tool (SREGym/clients/stratus/tools/submit_tool.py) expects status to be 200.

if result["status"] != "200":
        logger.info(f"HTTP submission failed: {result}")

@HacksonClark
HacksonClark marked this pull request as ready for review February 16, 2026 22:17
@yimingsu01

Copy link
Copy Markdown
Collaborator

@HacksonClark I'm testing this with a MS problem and a normal problem

@HacksonClark HacksonClark self-assigned this Feb 17, 2026
@HacksonClark

Copy link
Copy Markdown
Member Author

Closes #575

@yimingsu01

Copy link
Copy Markdown
Collaborator

@HacksonClark can confirm that the previous broken MS problem now can correctly finish (capacity_decrease_rpc_retry_storm)

@yimingsu01

Copy link
Copy Markdown
Collaborator

Closes #585

@yimingsu01

yimingsu01 commented Feb 20, 2026 •

Copy link
Copy Markdown
Collaborator

Testing three problems tonight to merge this.

  • capacity MS problem
  • one tidb problem
  • one normal old problem for regression

@yimingsu01 yimingsu01 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works

@yimingsu01
yimingsu01 merged commit 172d051 into main Feb 20, 2026
2 checks passed
@yimingsu01
yimingsu01 deleted the claude/issue-244-20260213-2244 branch February 20, 2026 06:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Submission API does not return HTTP response until other logic finishes

3 participants