Skip to content

.github: Run workflows on self-hosted triton runners - #424

Open
ncalvas wants to merge 2 commits into
trunkfrom
ci/self-hosted-runners
Open

ncalvas wants to merge 2 commits into
trunkfrom
ci/self-hosted-runners

Conversation

@ncalvas

@ncalvas ncalvas commented Sep 24, 2026

Copy link
Copy Markdown
Member

Summary

  • Run every job in .github/workflows/rebase.yml (features, illumos, macos) and .github/workflows/create-quarterly-feature.yml (create-features) on the self-hosted runners instead of GitHub-hosted ubuntu-latest
  • All four jobs now use runs-on: { group: triton, labels: [self-hosted, APP] }
  • Add a "Reset workspace left by previous runs" step before actions/checkout in each job

Context

The triton runner group has six self-hosted runners, three in DA3 and three in IAD39. All six carry the APP label, so any job can run on any of them.

GitHub-hosted runners give every job a fresh VM. Self-hosted runners keep the workspace between jobs, and these workflows assume a clean one. git remote add netbsd fails once the remote exists. A run that is cancelled (cancel-in-progress: true) or hits a rebase conflict leaves a rebase in progress, which blocks every later run on that runner.

The new step fixes this. If a .git directory exists, it aborts any rebase or merge in progress, removes the netbsd remote, and deletes .git/modules plus the extra, joyent and wip submodule checkouts. actions/checkout then cleans and resets the tree as usual.

Notes

  • Workspace reuse is a deliberate choice and can be revisited. We keep .git between runs rather than wiping the workspace each job. Later runs fetch only what changed instead of recloning the ~1.9 GB repo. Wiping every job, as GitHub-hosted runners effectively do, would be about 8 full clones an hour (~300 GB a day) into our DCs. The trade-off is that correctness depends on the reset step catching all leftover state. If that proves fragile, switch to wiping the workspace at the start of each job.
  • Not tested before merge. The first scheduled run after merge is the test. Watch the first few hourly Rebase branches against NetBSD runs, especially the second run on each runner, since that is the first one to reuse a workspace.
  • Runner group access: triton must allow this repo and must have "Allow public repositories" turned on, because pkgsrc is public. I couldn't check this without org admin. If jobs wait in the queue, this is the cause.
  • Public repo on self-hosted runners: these workflows only run on schedule and workflow_dispatch, so outside contributors can't trigger them. Do not add pull_request triggers without first requiring approval for outside contributors.
  • trunk is force-pushed every hour by the rebase job itself. Merge this promptly, or rebase the branch onto the latest trunk right before merging. Otherwise the PR base will have been rewritten under it. Prefer "Rebase and merge" so no merge commit lands on trunk.
  • Capacity: rebase.yml starts 6 features jobs at once, and there are 6 runners, so they fit with no spare.

@ncalvas ncalvas self-assigned this Sep 24, 2026
Move every job in rebase.yml and create-quarterly-feature.yml off
GitHub-hosted ubuntu-latest and onto the org's self-hosted runners in
group triton. APP is the one label shared by all six runners (three in
DA3, three in IAD39).

GitHub-hosted runners start each job on a fresh VM. Self-hosted runners
keep the workspace, and these jobs assume a clean one: git remote add
netbsd fails once the remote exists, and a cancelled or conflicted
rebase blocks every later run on that runner. Add a step before checkout
to abort any rebase or merge in progress, remove the netbsd remote and
drop the submodule checkouts. Keeping .git means later runs fetch only
what changed rather than the whole repository.
@ncalvas
ncalvas force-pushed the ci/self-hosted-runners branch 2 times, most recently from 240893c to 51c8c61 Compare September 24, 2026 15:12
@github-actions
github-actions Bot force-pushed the trunk branch 25 times, most recently from e32aab4 to 595be89 Compare September 26, 2026 16:31
@github-actions
github-actions Bot force-pushed the trunk branch 30 times, most recently from 8eb0399 to 90cae48 Compare October 1, 2026 05:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants