Repository navigation
ci(swebench): every smoke run fails with 'cloud model unavailable (HTTP 404)' #4869
Description
The SWE-bench smoke check fails on every PR that runs it, so it currently tells us nothing about the agent. All three tasks error in about 2 seconds with model backend failed: The selected cloud model is unavailable (HTTP 404), before the agent takes a step. It has not passed once since the FIREWORKS_API_KEY secret was added on 2026-10-04 (failed runs: 2026-10-06 00:27, and 08:52 on #4864–#4867).
The key and the model are fine: with that key, GET https://api.fireworks.ai/inference/v1/models lists accounts/fireworks/models/deepseek-v4p1-flash, and a direct chat completion to it answers. The break is between GAIA and Fireworks inside the job: the embedded Lemonade it starts reports "cloud": {"providers": []}, so fireworks.deepseek-v4p1-flash resolves to nothing.
Next step: register the Fireworks provider with the embedded Lemonade in eval_swebench.yml before gaia eval tasks run, the way the Agent UI and TUI do (src/gaia/ui/routers/providers.py, POST /api/providers/fireworks/connect), instead of relying on Lemonade reading LEMONADE_FIREWORKS_API_KEY from its environment as the workflow's header comment assumes.
🔍 Technical details
- Workflow:
.github/workflows/eval_swebench.yml;SWE_MODEL: fireworks.deepseek-v4p1-flash; the key reaches lemond asLEMONADE_FIREWORKS_API_KEY(job env), andchild_env()does not strip it (only GAIA's internal tokens are stripped). - Embedded Lemonade 2026.40.0,
gaia lemonade embedded status→Backends: none; no provider registration step appears in the log. - Example failing job: https://github.com/amd/gaia/actions/runs/37439029552/job/112187893380
- Not a required check, so it does not block merges.
Activity
github-actions commentedon Oct 6, 2026
The code backs this up. Lemonade only uses
LEMONADE_FIREWORKS_API_KEYfor a cloud provider that is already registered with it, and nothing registers Fireworks with the fresh embedded Lemonade in CI. That's why the provider list is empty and every request gets a 404.The fix doesn't need the Agent UI server. Add a step to
eval_swebench.ymlafter the embedded Lemonade starts that sends the same registration request the UI's/connectsends, straight to Lemonade. The env key should then be picked up without a separate auth call. Then check/system-infoand fail the step if Fireworks shows no discovered models. Without that check, a bad key or a missing provider looks like an agent regression again.🔍 Technical details
connect_provider(src/gaia/ui/routers/providers.py) runsPOST /installwith{"backend": "cloud", "provider": "fireworks", "wire_format": "openai", "base_url": FIREWORKS_BASE_URL, "auth_header_name": "Authorization", "auth_header_prefix": "Bearer "}, thenPOST /cloud/authonly when a key was pasted.ensure_authenticated(src/gaia/llm/cloud_keys.py) returnsFalsewhen/system-infohas no entry for the provider. It treatsenv_var_setas already authenticated, so once the provider is registered the job-env key should be enough. The readiness probe's key replay can't help here, because there's nothing to replay into.lemonade_auth_headers(resolve_lemonade_api_key())with the request, because the embedded instance is locked withLEMONADE_API_KEY. A shortpython -cthat reuses_registration("fireworks", …)keeps the payload in step with the UI and TUI (tui/internal/lemonade/cloud.go).POST /installmeans this Lemonade can't register cloud providers. The UI maps that to "update to 11.8.1+". I haven't checked 2026.40.0 against that floor, so the first run should confirm it.