Cirrus CI free usage going away - job runtime issues & credits

This is getting kinda annoying: [skip cirrus] is broken and the wheel build and other CI jobs are running way too often. It was probably broken by the addition of other logic like always triggering on PRs with build-related label.

What do you want to happen with the build label? I use it when I want to test wheel builds. It is a bit annoying, as simply adding a label will retrigger the builds.

Labels are for indicating what a PR or issue is about, so I add it to build-related PRs. And running wheel builds by default is fine, but the [skip cirrus] should supercede that (just like it does for [skip actions] on GHA).

but the [skip cirrus] should supercede

Fair enough, but some labels do trigger wheel builds, so they aren't only about what the PR or issue is about. Not sure how we could change that unless we want to add a [build wheels] option.

Why [skip cirrus] doesn't work is explained by

numpy/.cirrus.star

Lines 23 to 25 in 7fc7277

    
           # Obtain commit message for the event. Unfortunately CIRRUS_CHANGE_MESSAGE 
        
           # only contains the actual commit message on a non-PR trigger event. 
        
           # For a PR event it contains the PR title and description.

It looks like it's not the build label though, we run wheel builds on Cirrus always. E.g. look at this doc-only PR with no labels: gh-24277. This wastes a huge amount of compute time.

@andyfaff

Cirrus is a bit different. One problem is that it cannot be manually run, I think it needs something in the *.yml file for that. I also think (only) wheels are built and tested, I'm not sure how a build can be made without that. It is on my problem list, but not annoying enough to spend time on it. Maybe @andyfaff has some thoughts.

.cirrus.star is triggered to run with a lot of GH events on the numpy/numpy repo, e.g. if there are commits to PRs, commits to branches, PRs are opened, labels are attached to PRs, tags are pushed, merges, etc.

Here is the current logic for numpy's cirrus CI, as contained in .cirrus.star:

only run jobs only on the numpy/numpy repo.
(do a wheels build if a 'nightly' cron job is requested)
for all other triggers (and there can be a lot of them) the configuration script looks at the commit message of the SHA, provided by CIRRUS_CHANGE_IN_REPO, for the event. If there is [skip cirrus] or [skip ci] in the message then don't run any CI.
otherwise do the wheel build and the macosx_arm64 runs.

This logic will dictate that the wheels always get built if there is no [skip cirrus] in the commit message.

[skip cirrus] is broken and the wheel build and other CI jobs are running way too often.

I'm pretty sure [skip cirrus] works for commits to PRs. But for other events if the commit message for the SHA doesn't contain those words then the wheel build and macosx_arm64 will run. For example adding labels to a PR will trigger these runs if the last commit to the PR doesn't contain the magic words.

It's not clear to me from previous comments what additional logic is being requested. The following environment variables may be useful in reducing the number of runs made:

CIRRUS_PR | PR number if current build was triggered by a PR - we may be able to query Github as to what occurred in the PR to trigger cirrus-ci. We already query GH in cirrus.star.
CIRRUS_PR_LABELS | comma separated list of PR's labels if current build was triggered by a PR - examine the labels to see if 36 - build is present. e.g. if you look in the debugging info for https://cirrus-ci.com/build/4758553810960384 you'll see that label present.
CIRRUS_TAG | Tag name if current build was triggered by a new tag.

Possible extra logic that could be done:

don't automatically do wheel build unless ...
it's a nightly job
scan SHA message for [wheel build], do wheel build if present
scan CIRRUS_PR_LABELS for 36 - Build, do wheel build if present
build wheels for a tag event

Relevant links:

CIRRUS_CHANGE_IN_REPO
numpy's .cirrus.star config
numpy's wheel build config
numpy's macosx_arm64 config
cirrus environment variables

EDIT:
[skip cirrus] wasn't working, fixed in #24285.

Having just said that I see an issue at https://github.com/numpy/numpy/blob/main/.cirrus.star#L27. I'll open a PR.

You can examine what we requested from the GH API in 24282. The github request is:
https://api.github.com/repos/numpy/numpy/git/commits/cad8595a8c86c173285d82b61f6797ff24324364.

This is what is returned:

{
  "sha": "cad8595a8c86c173285d82b61f6797ff24324364",
  "node_id": "C_kwDOAA3dP9oAKGNhZDg1OTVhOGM4NmMxNzMyODVkODJiNjFmNjc5N2ZmMjQzMjQzNjQ",
  "url": "https://api.github.com/repos/numpy/numpy/git/commits/cad8595a8c86c173285d82b61f6797ff24324364",
  "html_url": "https://github.com/numpy/numpy/commit/cad8595a8c86c173285d82b61f6797ff24324364",
  "author": {
    "name": "Andrew Nelson",
    "email": "[email protected]",
    "date": "2023-07-29T00:15:13Z"
  },
  "committer": {
    "name": "Andrew Nelson",
    "email": "[email protected]",
    "date": "2023-07-29T00:15:13Z"
  },
  "tree": {
    "sha": "1c9f90fbbcff3162542b6663e0fe75a86e819bb4",
    "url": "https://api.github.com/repos/numpy/numpy/git/trees/1c9f90fbbcff3162542b6663e0fe75a86e819bb4"
  },
  "message": "CI: correct URL in cirrus.star [skip cirrus]",
  "parents": [
    {
      "sha": "422854fa8dc501e5fcbd713093fdee04e7e9ebb8",
      "url": "https://api.github.com/repos/numpy/numpy/git/commits/422854fa8dc501e5fcbd713093fdee04e7e9ebb8",
      "html_url": "https://github.com/numpy/numpy/commit/422854fa8dc501e5fcbd713093fdee04e7e9ebb8"
    }
  ],
  "verification": {
    "verified": false,
    "reason": "unsigned",
    "signature": null,
    "payload": null
  }
}

The cirrus CI didn't run because [skip cirrus] is in dct['message'].

I think we should be able to limit the wheel build to whatever is in the GHA wheels build, if that's desired.

EDIT: apart from manual trigger, not sure how to do that.

@andyfaff

Thanks for the fix @andyfaff!

Given the major reduction in free resources available per 1 Sep (see https://cirrus-ci.org/blog/2023/07/17/limiting-free-usage-of-cirrus-ci/), I think we have a lot more work to do here unfortunately (and may consider buying some credits).

Regarding the label-based trigger, I think there are two things wrong with it:

the Build label is wrong for this, it should be a dedicated and clearly named label like trigger-cirrus
it's a problem that this label, once added to a PR, tends to stay on it and then the wheel builds run for every subsequent push. typically what's intended is a one-off "check wheel builds". It doesn't really make sense (resource-usage wise) to run the full battery of wheel builds on every push to a PR.

Given the above and that our resource usage at the current rate (see screenshot below) is completely unsustainable and would run at ~$2,500/month (or ~$1,400 after the upcoming price reductions also announced in the blog post linked above) if we'd have to pay for it from 1 Sep, I'd much prefer to get rid of label-based triggering completely. Manual wheel build triggers should be rare and reserved to maintainers who know what they are doing and are able to push an empty commit with the correct commands in the commit message.

CPU usage is also bad on cibuildwheel jobs (and note that credits go per CPU-minute, i.e. per core rather than per job); we need to ensure to use 2 cores for pytest:

I'll note that on jobs with 2 CPUs, using -n auto also isn't great, since pytest-xdist translates that to -n4 rather than -n2 and its scaling of parallelism is terrible and performance improvement tends to get negative from 4 jobs already. -n2 on 2-core jobs is already far from linear, 3 is getting questionable but still gains typically, >=4 quickly decreases performance.

Example log from a recent macos_arm64_test run showing we get 4 pytest-xdist workers:

$ /Users/admin/numpy-dev/bin/python3.10 -m pytest --rootdir=/private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/build-install/usr/lib/python3.10/site-packages -n auto -m 'not slow' numpy
============================= test session starts ==============================
platform darwin -- Python 3.10.6, pytest-7.4.0, pluggy-1.2.0
rootdir: /private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/build-install/usr/lib/python3.10/site-packages
configfile: ../../../../../pytest.ini
plugins: hypothesis-6.82.0, xdist-3.3.1
created: 4/4 workers
4 workers [34198 items]

Here is the full list of jobs and runtimes for a single run:

That's a total of ~222 CPU minutes for wheel builds per run, divided in

77.5 * 2 = 144 min on Linux aarch64 = $0.145 per run
33.5 * 4 = 134 min on macOS = $0.67 per run

So each wheel build costs about $0.80 each time it's triggered - this is a lot. We also have issues with some tests in the full test suite that need fixing (e.g., the slow typing tests shouldn't be run by default, they're the same on all platforms and take well over a minute). But most importantly, we should not be triggering wheel builds so much, they're only very rarely useful.

As you can see in gh-24289, that PR - which only tweaked a code comment in a meson.build file and I added the 04 - Documentation label to - still triggers a full set of wheel builds on Cirrus (I cancelled them manually after they started running).

And then after a merge to main, it's running yet again: https://cirrus-ci.com/build/5871472732798976.

This is the relevant code in .cirrus.star:

    # Obtain commit message for the event. Unfortunately CIRRUS_CHANGE_MESSAGE
    # only contains the actual commit message on a non-PR trigger event.
    # For a PR event it contains the PR title and description.
    SHA = env.get("CIRRUS_CHANGE_IN_REPO")
    url = "https://api.github.com/repos/numpy/numpy/git/commits/" + SHA
    dct = http.get(url).json()
    # if "[wheel build]" in dct["message"]:
    #     return fs.read("ci/cirrus_wheels.yml")

    if "[skip cirrus]" in dct["message"] or "[skip ci]" in dct["message"]:
        return []

    # add extra jobs to the cirrus run by += adding to config
    config = fs.read("tools/ci/cirrus_wheels.yml")
    config += fs.read("tools/ci/cirrus_macosx_arm64.yml")

    return config

I don't see any label-based triggers, also not in tools/ci/cirrus_*. I think what we need here is to (a) uncomment the lines with [wheel build], and (b) delete the line config += fs.read("tools/ci/cirrus_wheels.yml").

@rgommers

@rgommers, see #24286

We're at $0 now. There's a couple of issues:

Credits consumption went too fast. We did not use 99 credits in the last 35 days, more like 40.
SciPy has the opposite issue, zero credits were subtracted in the last 6 weeks. Since I paid both with the same credit card, it looks like the SciPy credit usage got subtracted from the NumPy funds 🤔.
I cannot add more credits right now, somehow Cirrus is not liking my credit card anymore.

I have to follow up with them, but in the meantime CI jobs may stop running.

I worst comes to worst, I also have a credit card 😉

I've added $98 today (20 Dec '23), they're accepting my credit card again. Don't know what happened, assuming some validation issue. I still need to follow up with Cirrus CI about issues with mixing NumPy/SciPy credits and getting better invoices.

Hmm, with all the recent changes to wheel builds we're now up to about $2.75 per run, that's a lot more than higher up.

Reasons:

We added musllinux_aarch64, doubling the duration of the Linux jobs
We added double the number of macOS jobs, for Accelerate wheels
The test suite has gotten slower by a significant amount

Only the last reason is fixable.

In anticipation of lots of wheel builds in the run-up to 2.0.0rc1 I added $121 more credits - we're at 164 credits as of today (7 Jan 2024). Should be enough until sometime in Feb. Next up: testing the reimbursement process.

Bought another $97 in credits (note: slightly different amount each time is to ensure the invoices are easier to tell apart). With a number of the macOS arm64 jobs now migrated to GHA, this should hopefully last us a little while.

Just a note on support quality: I noticed that a single job ran for ~17 minutes longer yesterday, and hence consumed more credits. I emailed Cirrus support; they were already on it, and fixed the issue with a comprehensive explanation within an hour or so, with credits returned and all previous jobs in the last month also audited. Impressive.

@rgommers

@rgommers This is an example that supports the argument some economists make for leaving spending decisions to the individual :)

We hit zero; I added $150 with the shiny new NumPy credit card - seems to have gone fine.

Looks like we're burning a bit more with recent release and dev activity - but still well within budget:

Added another $151 today. Burn is roughly $50/month at the moment.

With the free usage months gone, the graph is more informative now:

We're down to a burn of $10/month, almost all for FreeBSD. This has become quite easy to keep under control, I think we're happy as is. So I will close this issue.

From Cirrus Labs to join OpenAI:

Cirrus CI will shut down effective Monday, June 1, 2026.

It is used only in the FeeBSD run. I guess we will have to forego that.

Yeah I had a look at it, no good replacement for FreeBSD. We don't really need that job, so we'll just go back to what we had before, which is no specific coverage for building on FreeBSD.

Navigation Menu

Repository navigation

Cirrus CI free usage going away - job runtime issues & credits #24280

Relevant links:

17 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	# Obtain commit message for the event. Unfortunately CIRRUS_CHANGE_MESSAGE
	# only contains the actual commit message on a non-PR trigger event.
	# For a PR event it contains the PR title and description.

Cirrus CI free usage going away - job runtime issues & credits #24280

Description

Activity

charris commented on Jul 28, 2023

rgommers commented on Jul 28, 2023

charris commented on Jul 28, 2023

rgommers commented on Jul 28, 2023

charris commented on Jul 28, 2023

andyfaff commented on Jul 29, 2023

Relevant links:

andyfaff commented on Jul 29, 2023

andyfaff commented on Jul 29, 2023

andyfaff commented on Jul 29, 2023

rgommers commented on Jul 30, 2023

rgommers commented on Jul 30, 2023

rgommers commented on Jul 30, 2023

rgommers commented on Jul 30, 2023

andyfaff commented on Jul 30, 2023

17 remaining items

rgommers commented on Nov 16, 2023

charris commented on Nov 16, 2023

rgommers commented on Dec 20, 2023

rgommers commented on Dec 22, 2023

rgommers commented on Jan 7, 2024

rgommers commented on Mar 8, 2024

rgommers commented on Mar 8, 2024

charris commented on Mar 8, 2024

rgommers commented on Jun 5, 2024

rgommers commented on Aug 11, 2024

rgommers commented on Sep 10, 2025

DimitriPapadopoulos commented on May 2, 2026

mattip commented on May 2, 2026

rgommers commented on May 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions