Repository navigation
Nightly flagship eval did not pass #4870
Description
The nightly flagship eval on main did not pass.
- Run: https://github.com/amd/gaia/actions/runs/37437246150
- Scenario lanes:
cancelled - Outcomes (
gaia eval tasks):cancelled
Start with the run's summary page: it names which lanes measured, which measured nothing, and
how long each took. Then download the failing lane's eval-<lane> artifact - it carries the
transcripts, judge rationales, backend log and Lemonade logs. Those machines are ephemeral and
already destroyed, so the artifact is the only copy, and it expires in 14 days.
A nightly failure with no PR behind it usually means the environment moved, not the code: a
Lemonade version bump, a re-quantised model, a driver or runner-image change. PR runs are
path-filtered and cannot see any of that, which is why this run exists.
cancelled rather than failure means no measurement happened at all - a manual cancel, or a
lane that waited out GitHub's 24h queue limit. Re-dispatch the workflow.
Activity
github-actions commentedon Oct 7, 2026
The nightly flagship eval on
maindid not pass.failuregaia eval tasks):successStart with the run's summary page: it names which lanes measured, which measured nothing, and
how long each took. Then download the failing lane's
eval-<lane>artifact - it carries thetranscripts, judge rationales, backend log and Lemonade logs. Those machines are ephemeral and
already destroyed, so the artifact is the only copy, and it expires in 14 days.
A nightly failure with no PR behind it usually means the environment moved, not the code: a
Lemonade version bump, a re-quantised model, a driver or runner-image change. PR runs are
path-filtered and cannot see any of that, which is why this run exists.
cancelledrather thanfailuremeans no measurement happened at all - a manual cancel, or alane that waited out GitHub's 24h queue limit. Re-dispatch the workflow.
github-actions commentedon Oct 9, 2026
The nightly flagship eval on
maindid not pass.failuregaia eval tasks):successStart with the run's summary page: it names which lanes measured, which measured nothing, and
how long each took. Then download the failing lane's
eval-<lane>artifact - it carries thetranscripts, judge rationales, backend log and Lemonade logs. Those machines are ephemeral and
already destroyed, so the artifact is the only copy, and it expires in 14 days.
A nightly failure with no PR behind it usually means the environment moved, not the code: a
Lemonade version bump, a re-quantised model, a driver or runner-image change. PR runs are
path-filtered and cannot see any of that, which is why this run exists.
cancelledrather thanfailuremeans no measurement happened at all - a manual cancel, or alane that waited out GitHub's 24h queue limit. Re-dispatch the workflow.
github-actions commentedon Oct 10, 2026
The nightly flagship eval on
maindid not pass.failuregaia eval tasks):successStart with the run's summary page: it names which lanes measured, which measured nothing, and
how long each took. Then download the failing lane's
eval-<lane>artifact - it carries thetranscripts, judge rationales, backend log and Lemonade logs. Those machines are ephemeral and
already destroyed, so the artifact is the only copy, and it expires in 14 days.
A nightly failure with no PR behind it usually means the environment moved, not the code: a
Lemonade version bump, a re-quantised model, a driver or runner-image change. PR runs are
path-filtered and cannot see any of that, which is why this run exists.
cancelledrather thanfailuremeans no measurement happened at all - a manual cancel, or alane that waited out GitHub's 24h queue limit. Re-dispatch the workflow.
github-actions commentedon Oct 12, 2026
The nightly flagship eval on
maindid not pass.failuregaia eval tasks):successStart with the run's summary page: it names which lanes measured, which measured nothing, and
how long each took. Then download the failing lane's
eval-<lane>artifact - it carries thetranscripts, judge rationales, backend log and Lemonade logs. Those machines are ephemeral and
already destroyed, so the artifact is the only copy, and it expires in 14 days.
A nightly failure with no PR behind it usually means the environment moved, not the code: a
Lemonade version bump, a re-quantised model, a driver or runner-image change. PR runs are
path-filtered and cannot see any of that, which is why this run exists.
cancelledrather thanfailuremeans no measurement happened at all - a manual cancel, or alane that waited out GitHub's 24h queue limit. Re-dispatch the workflow.
github-actions commentedon Oct 13, 2026
The nightly flagship eval on
maindid not pass.failuregaia eval tasks):successStart with the run's summary page: it names which lanes measured, which measured nothing, and
how long each took. Then download the failing lane's
eval-<lane>artifact - it carries thetranscripts, judge rationales, backend log and Lemonade logs. Those machines are ephemeral and
already destroyed, so the artifact is the only copy, and it expires in 14 days.
A nightly failure with no PR behind it usually means the environment moved, not the code: a
Lemonade version bump, a re-quantised model, a driver or runner-image change. PR runs are
path-filtered and cannot see any of that, which is why this run exists.
cancelledrather thanfailuremeans no measurement happened at all - a manual cancel, or alane that waited out GitHub's 24h queue limit. Re-dispatch the workflow.
github-actions commentedon Oct 14, 2026
The nightly flagship eval on
maindid not pass.failuregaia eval tasks):successStart with the run's summary page: it names which lanes measured, which measured nothing, and
how long each took. Then download the failing lane's
eval-<lane>artifact - it carries thetranscripts, judge rationales, backend log and Lemonade logs. Those machines are ephemeral and
already destroyed, so the artifact is the only copy, and it expires in 14 days.
A nightly failure with no PR behind it usually means the environment moved, not the code: a
Lemonade version bump, a re-quantised model, a driver or runner-image change. PR runs are
path-filtered and cannot see any of that, which is why this run exists.
cancelledrather thanfailuremeans no measurement happened at all - a manual cancel, or alane that waited out GitHub's 24h queue limit. Re-dispatch the workflow.