Restructuring GitHub Actions CI jobs

These CI jobs are hard to navigate with several levels of indirection (a workflow file, then an action file, then two tools/travis-*.sh scripts), and now that we have to move a lot of jobs over to Meson it's a good time to clean up. The build_test.yml is particularly messy, and contains 22 jobs which is too much (quite a few are unnecessary too).

A few thoughts:

The windows_meson.yml and windows_clangcl.yml can be merged, and named windows.yml
It's useful to keep unusual / niche platform jobs in separate files. So let's keep emscripten.yml, cygwin.yml, linux_musl.yml and linux_compiler_sanitizers.yml.
Let's break up build_test.yml:
- linux_simd.yml: all the SIMD related jobs (~8 jobs) and the old_gcc one
- linux_blas.yml: all the BLAS/LAPACK related variations we're testing (should end up with ~6 jobs)
- linux.yml: testing across Python versions (PyPy, python-dbg, a pre-release Python in summer time), build-via-sdist, run benchmarks, measure code coverage, and other build options like relaxed-strides. This will also fold in linux_meson.yml.
The SIMD tests are the only ones that may use a .sh script, for the checks that are now in travis-test.sh. For the rest, those shell script are just hard to read cruft.
I think we'd like a summary of what the larger files to at the top. In particular for SIMD, BLAS and the main linux.yml ones, that would be really helpful.
No more matrix'ed tests on a single variable, that's too much repetition of the exact same thing. We must be testing Python 3.9 + OpenBLAS + ubuntu-latest with its default GCC version more than 5 times by now, and it doesn't do much except heat the planet.

Sounds good. One downside of breaking up the jobs is how to do a smoke test run before starting many other runs. I don't know how we could pause all other CI runs until it passes, extra points for pausing cirrus travis, and azure. It would be too disruptive to do that with a local precommit-hook before allowing the PR to be pushed, but is there such a thing for the repo?

Another thing that would be nice is to be able to label a PR [docs only] if it only changes the rst documentation, or [smoketest only].

There is a persistent failure in travis, we should mark that as "not fail".

One downside of breaking up the jobs is how to do a smoke test run before starting many other runs

That's not too difficult, I think smoke tests can be run for the 3 main files (simd, blas, linux). I had thought of that already, will keep it as a required feature for now.

I don't know how we could pause all other CI runs until it passes, extra points for pausing cirrus travis, and azure

Controls between different CI providers are a pain, and I think an orthogonal topic to GHA changes. I'd much prefer not to think about that here.

There is a persistent failure in travis, we should mark that as "not fail".

We should turn TravisCI off for anything but SIMD PRs.

We discussed this in the community meeting today. Overall sentiment was that:

this refactor and breaking up build_test.yml is a good idea,
it would be nice to make https://github.com/numpy/numpy/actions/ more easy to navigate (lots of things in the sidebar and in "dummy" jobs like the CircleCI Artifact Redirector)
it'd be useful to combine some builds into one job (as done in some SIMD jobs already) in order to reduce the number of separate jobs. that has the advantage of not only making things easier to find, but that fewer things get run when there's a failure
we can remove all the numpy.distutils based jobs as soon as possible, no need to wait
we can use QEMU for some ARM NEON SIMD testing, no need for Cirrus CI. QEMU should be fast enough here (and we already have an armv7 QEMU job
ideally, if we can trim things down enough, we can get rid of Azure completely

CI: split `build_test.yml` into three GHA jobs files

Here are the current workflows on github actions and the time they take to run (after merging #24493 and #24479)

Workflow	time	Runs on merge to main
BLAS tests (Linux)	9-14 min	no
CircleCI artifact redirector	10 sec	no
CodeQL	5-7 min	yes
Dependency Review	< 20s	no
Linux Qemu tests	20-25 min	no
Linux tests	19-30 min	yes
Pull Request Labeler	<20s	no
Run MyPy	6-14 min	no
Scorecards supply-chain security	1 min	runs only on merges
SIMD tests (Linux)	23-26 min	no
Test Address Sanitizer Build (Linux)	19 min	see PR #24208
Test Emscripten/Pyodide build	10-15 min	no
Test muslinux ×86 64	9-10 min	no
Test on Cygwin	34-50 min	no
Test with compiler sanitizers (Linux)	21-32 min	no
Wheel builder	20 min	mostly skipped, runs on a schedule
Windows tests	14-17 min	no

In addition we have the

azure workflow that takes ~34-50 min
the cirrus builds (two jobs: ~12 minutes on aarch64 linux and ~3 minutes on macos-arm64) and
the circleCI workflow that takes ~23 min

With GH Actions it should be possible to make a workflow that triggers other workflows. One could imagine a main Linux workflow (the wheel builder config?) that takes on the responsibility of doing the builds, that are then used by subsiduary workflows that do the different test runs.

that takes on the responsibility of doing the builds, that are then used by subsiduary workflows that do the different test runs.

I think the vast majority of builds should be for different build configs. We're running the whole test suite, there's not much need for retesting the same wheel multiple times.

The two most expensive files have a smoketest that acts as a gate for the other jobs in that file. I think we can further improve on this, but it requires a bit of design/discussion on how to do that. Ideally after removing all of setup.py.

I have a decent idea of what's still left for setup.py removal, but I'll open a PR to verify, and then hopefully we can parallelize the work on that between a bunch of us. I'll try and post a check list within a few hours.

@ev-br

Okay, here are all jobs that need work, based on the failing builds from gh-24519:

CircleCI docs build (release notes issue, ~~probably easy to resolve~~)
TravisCI (can be removed)
Azure: Linux conda job
Azure: macOS jobs (2x, py39, 32/64-bit BLAS)
GHA Linux tests (3x, basic/debug/benchmark)
GHA SIMD tests (Linux) - 4 jobs
GHA BLAS tests (Linux) - openblas64_setuppy (can be removed)
GHA Cygwin job
GHA Emscripten job

@ev-br will have a look at the Emscripten job. @mkoeppe would you be able to have a look at the Cygwin job? @seiko2plus could you look at the SIMD tests (Linux) jobs? @mattip there is still a problem with the PyPy 3.9 on Windows job in Azure, would you be able to take that one? I can have a look at the conda and macOS Azure jobs and the two to-be-removed jobs. Any takers for the CircleCI job or the GHA Linux tests jobs?

After all that is done, we also still have jobs to add:

32-bit Windows wheel builds
testing against different BLAS libraries (added in ENH: meson: implement BLAS/LAPACK auto-detection and many CI jobs #24893)

plus it'll be easier to clean up the docs if all the setup.py stuff is really gone. Almost there 🤞🏼

I keep on finding special-case test bits, like this one:

        python -m pip install vulture sphinx==4.3.0 numpydoc==1.4.0 ninja
      displayName: 'Install dependencies; some are optional to avoid test skips'
    - script: /bin/bash -c "! vulture . --min-confidence 100 --exclude doc/,numpy/distutils/ | grep 'unreachable'"
      displayName: 'Check for unreachable code paths in Python modules'

We should keep all of those only on GitHub Actions, because they're much easier to debug there. And we should have them all together in a separate job, not mixed into some random job (macOS tests on Azure in this case).

I believe we can get rid of Azure completely. It's mostly Windows jobs that are left after gh-24520 is merged. For Windows, we now have the following:

MSVC, 32-bit Python 3.10, no BLAS (GHA)
MSVC, 64-bit Python 3.11, 32-bit OpenBLAS (GHA)
Clang-cl, 64-bit Python 3.11, 32-bit OpenBLAS (GHA)
MSVC, 64-bit Python 3.11, 64-bit OpenBLAS (Azure)
MSVC, 64-bit Python 3.10, 32-bit OpenBLAS (Azure)
MSVC, PyPy 3.9, 64-bit OpenBLAS (but disabled, cause broken) (Azure)

We only need 4 jobs, to cover:

compilers: MSVC, Clang-cl
Python interpreters: 32-bit CPython, 64-bit CPython (min/max supported versions), PyPy 3.9
OpenBLAS: no BLAS, LP64 OpenBLAS, ILP64 OpenBLAS

This can be a sparse test matrix, there's no need for a dense one.

I'll take the CircleCI one as well.

Getting closer - I'll take GHA Linux tests, (3x, basic/debug/benchmark) as well.

Another thing that would be nice is to be able to label a PR [docs only] if it only changes the rst documentation, or [smoketest only].

It's possible to set an Action to run only if certain files are modified; this example looks like a decent starting place for not running the main tests on a PR that only changes documentation, or configuration for other CI services.

Unfortunately, it doesn't signal other CI services that they don't need to run either.

MAINT: Split up .github/workflows to match main

@rgommers

@rgommers I think this can be closed?

Yes, this is pretty much all done. We haven't killed the last Azure DevOps jobs yet, but they don't seem too troublesome right now. We can always circle back to this later if needed.

A final note on this issue, since here is where we discussed getting rid of Azure DevOps: that goal was achieved in gh-30232.

Navigation Menu

Repository navigation

Restructuring GitHub Actions CI jobs #24410

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Restructuring GitHub Actions CI jobs #24410

Description

Activity

mattip commented on Aug 16, 2023

rgommers commented on Aug 16, 2023

rgommers commented on Aug 16, 2023

mattip commented on Aug 24, 2023

andyfaff commented on Aug 24, 2023

rgommers commented on Aug 24, 2023

rgommers commented on Aug 24, 2023

rgommers commented on Aug 25, 2023

rgommers commented on Aug 25, 2023

rgommers commented on Aug 25, 2023

rgommers commented on Aug 27, 2023

DWesl commented on Sep 7, 2023

mattip commented on Jan 27, 2024

rgommers commented on Jan 30, 2024

rgommers commented on Nov 17, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions