Repository navigation
Conversation
|
I should add, the TianoCore/EDK2 open source firmware stack includes a port of MicroPython, but this was forked from MicroPython in 2018 and widely diverges from the current mainline. Part of the intention here is to have upstream for UEFI in the mainline code. |
|
Related old issue: #7053 |
This PR introduces a new port to allow MicroPython to run on a PC under the UEFI firmware, prior to a full OS booting. While PCs are not microcontrollers, the pre-boot firmware environment has similarities both in terms of the limited platform and the direct access hardware (since the code runs in "Ring 0"). The port supports both Intel (x64) and ARM (aa64) UEFI platforms. The UEFI port provides comprehensive wrappers around most of the UEFI API surface. This allows Python code to discover and access hardware, read and write firmware variables and settings, directly interact with extensible EFI protocols and manage the boot process, up to and including implementing custom OS bootloaders in Python. The port provides access to firmware networking on machines which support this, using the standard MicroPython `network` module. Note that this initial port lacks support for Wi-Fi pending hardware and drivers for testing this. TLS support can be enabled either using mbedTLS or by connecting to the EFI_TLS protocol if the host's firmware provides this. The choice is compile-time; using EFI_TLS saves 190KB of 610KB but not all network-enabled firmwares support this. NOTE: The build process for this port makes extensive use of Docker to handle the cross-platform nature of the port. Some of the code here was typed by Claude, but the architecture, design and planning was entirely human! Signed-off-by: Nicko van Someren <[email protected]>
|
19k lines is way too much for a single commit or to review at once! And I didn't see any tests added, but maybe I missed them? I would expect them to run on CI. But of course before doing work in that direction, we should agree this is something we want in MicroPython in the first place. |
|
@dlech Yes, 19K lines is a lot, but there is a lot of new functionality here. This is both a port to make MicroPython run on a new platform and extensive support for the runtime on that platform. I squashed the commits because every single time I've submitted a multi-commit PR to MicroPython in the past I've been asked to squash them into one. This commit includes extensive tests which can be run automatically (they are a conditional component of The code takes pains to make no changes whatsoever to I confess that I don't know enough about the CI system for me to add tests there, but I'm open to suggestions. |
|
As a comparison with another PR that adds a new port: PR #18910 is 18.5k lines, adds basic REPL, UART and Pin capabilities, and is almost ready for merging. |
d37d012 to
9db2523
Compare
The `network` module now surfaces Wi-Fi interfaces when present and allows them to be configured as a client. Note that the EFI Wi-Fi protocols do not support configuring interfaces as APs, so this functionality is unavailable on the `uefi` port. Signed-off-by: Nicko van Someren <[email protected]>
|
I had a chance to test the Wi-Fi support, so I have now pushed a commit with wireless networking (and updated documentation to match). Note that the EFI wireless networking protocol doesn't support AP mode, so nor does this port. |
|
On top of docs domments, I think ports/uefi/TODO.md also uses an awful lot of words to say not a lot (including some very obvious Claude tells- honestly, why is everything a story?). Effectively there's no native emitter because there is no native emitter. That change might be completely orthogonal to this port? (used by, but no co-dependent upon?) Looks like the test harness and fixtures got lumped in with the port, too, I had to be very explicit about getting those folded into the right places, using the right method (took a couple tries). Should probably be in /tools and /tests with reasoning for why the existing harnesses and methods don't apply (if they can't apply). I think separate commits make sense where they address separate concerns- ie one for docs, one for the port, one for tests. Often those are useful for reviewing, but not really necessary for merging- thus a last-minute squash. (note: I don't know much about eufi, but I've been fighting Claude to get large changes into the right shape for upstream lately) note: open a new prompt and: "Review this uefi port. How closely does it adhere to the explcit and implicit standards of the rest of the micropython codebase? Lay out some suggestions for improvement." edit: Just took my own advice on one of my own PRs (remembering to ask the right question it is half the battle) and got served a sizeable list of fixes 😆 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19432 +/- ##
========================================
Coverage 98.51% 98.52%
========================================
Files 177 180 +3
Lines 22927 23244 +317
========================================
+ Hits 22586 22900 +314
- Misses 341 344 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Code size report: |
10cba3d to
1e7ad16
Compare
A new test infrastructure in ports/uefi/tests has been added to allow running of the standard MicroPython test suite on QEMU-emulated instances running UEFI firmware. All standard tests that can be run on the headless UEFI instances have been enabled. The original, hard-wired bring-up test infrastructure has been removed. All previous Docker-based tests that fit within the standard testing framework have been ported and moved to tests/ports/uefi. The test coverage has been expanded to cover the vast majority of the EFI API surface. A new CI workflow has been added in .github/workflows/ports_uefi.yml to perform full test suite execution through GitHub actions. The time and date handling has been reworked to run the real datetime as used by machine.RTC and the time()/time_ns()/localtime() off the same clock. Real time is read at startup and synced if the user makes a change, but all API calls use fast CPU timers as the time basis in normal operation to avoid disparities between "real time" and "CPU time" APIs. Note that the machine_timer test is known to have load-sensitive flakiness when run under emulation. Its failure here causes a warning rather than a test suite failure. The MICROPY_CONFIG_ROM_LEVEL setting has been moved to the "extra features" level which enabled a handful of features that were previously not on by default. A handful of bugs and missing implementations revealed by the more comprehensive testing have been fixed. Signed-off-by: Nicko van Someren <[email protected]>
|
I've stripped out the embedded bring-up testing, built out a test framework for running the standard test suite on MicroPython on UEFI on QEMU emulation and added a stack of port-specific tests to test everything that I can under emulation. I expanded the standard modules to include "extra features" and fixed a handful of integration points that stemmed from those. I've also added CI integration for all of this and it all passes. In addition to the test/CI work above, I think I've address @Gadgetoid's comments on the documentation and port TODO.md file. |
|
Making progress, but you'll forgive me for fighting fire with fire here. It's a lot of code in which I can trivially spot a lot of things that don't seem right, but can't always elucidate why. What follows is a guided code review using Claude and focussing on some of the things that stuck out to me in particular, plus some things it manifested on its own. It's an unholy wall of text but comparing a read of it with my spelunking into this code I broadly agree on points I understand, and concede on points I do not (efi/eufi ambiguity may have some reason I don't fathom). On top of this I'd say, why
for i in range(len(data)):
dst[i] = data[i]Is
Anyway I got distracted finding things (could follow-up with a Python code review if it's handy) , here's what Claude had to say (note this is pretty surface level stuff, but it takes a lot of effort to coax it deeper): UEFI port, review recommendationsA review of The port itself is strong: the C is well-sectioned and heavily commented, the hard Priorities: P1 blocks a clean upstream read; P2 is a clear improvement; P3 is 1. Namespaces & naming1.1 Collapse the
|
| File | Total lines | License | Actual code |
|---|---|---|---|
qstrdefsport.h |
28 | 28 | 0 |
include/alloca.h |
33 | 33 | 0 |
include/winsock2.h, ws2tcpip.h |
32 | 32 | 0 |
include/unistd.h |
33 | 32 | 1 |
uefi_stubs.c |
46 | 39 | 7 |
modtime_uefi.c |
47 | 38 | 9 |
mphalport.h |
55 | 46 | 9 |
This diverges from upstream in two demonstrable ways: upstream's own extmod/asyncio/*.py
carry no per-file MIT block (pure-Python modules rely on top-level LICENSE), and
upstream has begun adopting SPDX short-form (extmod/mbedtls/mbedtls_alt.c).
5.1 Adopt SPDX one-liners port-wide (P2)
Replace the 25-line block with SPDX-License-Identifier: MIT + a single copyright line.
Keeps attribution and license unambiguous, matches upstream's direction, and removes
~1,400 lines of boilerplate across the port.
5.2 Don't header trivial files at all (P2)
Trivial files should not carry license bloat - see
https://github.com/micropython/micropython/blob/master/ports/rp2/boards/manifest.py
(a manifest.py with no header). Apply the same to this port's manifest.py and the
one-line include/ shims.
5.3 Delete qstrdefsport.h if it defines no qstrs (P3)
It is currently 100% header + a "none yet" comment and zero code. If the Makefile's
QSTR_DEFS needs a target it can be satisfied without a committed all-boilerplate file.
5.4 Consolidate the include/ shims (P3)
Thirteen separately-headered files, several empty or one #define, is more surface than the
problem needs. Merge the zero/one-line shims where sensible and give each a terse one-line
"why this shim exists" note instead of the full block (see also 6.2).
6. Documentation & orientation
6.1 Add a file-by-file map (P1 - highest-value single change)
The README "Layout" section is one dense paragraph; the directory presents ~40 top-level
files with no index. Add a table: each C file -> one-line responsibility (uefi_event.c =
TPL/ISR-to-scheduler bridge + serial REPL; uefi_vfs.c = VFS over EFI_SIMPLE_FILE_SYSTEM;
main.c = entry / heap / stack / argv; etc.). This is what lets someone assess the port
without knowing UEFI.
6.2 Document the include/ shim surface in one place (P2)
The *-unknown-windows clang triple is deeply non-obvious and is why there is a
windows.h / winsock2.h / ws2tcpip.h shim. Add one paragraph - "we target a Windows PE
triple, so the toolchain expects these libc/Win headers; here is the whole shim list and
why" - near the -Iinclude line in the Makefile or an include/README. Tie
uefi_stubs.c's __chkstk / _fltused to the same triple story.
6.3 De-agent the README and fix the dangling reference (P1)
README.md:7 points readers to CLAUDE.md as "the agent operating manual", but CLAUDE.md
is .gitignored - upstream readers get a dead reference. Phrases like "the ABI landmine,
what-not-to-touch" read as internal tooling notes. Remove the CLAUDE.md reference and fold
any genuinely necessary build guidance into the README / Makefile comments.
6.4 Keep limitations DRY (P3)
README "Limitations" and TODO.md overlap. After 2.6 removes TODO.md, ensure the README
states current capability once.
Suggested execution order
- Subtractions first (fast, high signal): delete 2.1 debugging rigs, 2.6
TODO.md,
5.3qstrdefsport.h; fix 6.3 README /CLAUDE.md. - Namespace + naming: 1.1 collapse
efi/uefi, 1.2 renamemod*_uefi.c. - Relocations: 2.4 quarantine Docker/firmware into
dev/, 2.5 samples ->
examples/uefi/, 2.2/2.3 tidyscripts/anddocker/. - License sweep: 5.1 SPDX one-liners, 5.2 drop headers on trivial files.
- Test infra: 3.1 move toward standard runner, 3.2 narrow
harness.py. - Orientation docs: 6.1 file map, 6.2 shim index.
- Polish: 4.1
.gitignore, remaining P3s.
Uh, me again, the TLDR is that the more you delete/factor out, and the more you adhere to MicroPython idioms the easier this becomes to review. Right now the superficial quality of the Python code (duplicate modules, bytewise copies in lieu of slices, manual byte unpacking in lieu of struct, copy-pasted functions littered over the codebase, searches over globals() for guids, no const() use) is quite confusing; why put MicroPython in UEFI if you don't want to write MicroPython??
|
@Gadgetoid Many thanks for the feedback. I will work through these finding (by hand) and address your (and Claude's) suggestions. |
The changes to the license text removed just over 2,100 lines of boilerplate and touched 90 files, so they are being commited separately to avoid obscuring more subtle changes addressing the other points. Signed-off-by: Nicko van Someren <[email protected]>
The gzip.py file has been removed. I was not quite unused, since it was getting frozen and included in each build, but it was not needed. The ssl.py file replicated the functionality of the ssl.py in micropython-lib, wrapping the modern TLS support for backward compatibility. We now use the version on micropython-lib. The use of a 64 bit value in status.py if both deliberate and necessary, since this is the size of the return code from UEFI and bit 63 is meaningful. This has been left as it was. The hangover from when byte block slicing didn't work has been cleaned up. GUID packing/unpacking has been cleaned up (some of the packing wasn't even needed). I also revamped the whole registry system so that we build the canonical registry map and then fold it into the globals rather than trying to extract the map from the globals. UTF-16 support has been refactored into its own file and a single version is used throughout. The `efi` module has been dropped and the `efi.Timer` class has been relocated. The module file naming has been made both more internally consistent and more consistent with the naming conventions used in other ports. The README.md file now includes a structured map of the file layout. Much scaffolding, including all the bring-up network test scripts, have been removed. Signed-off-by: Nicko van Someren <[email protected]>
f4c1e52 to
2d9ef05
Compare
Testing the UEFI target now has run-test.py calling a target-specific harness, not the other way around. This required changes to tests/test_utils.py since the process of starting QEMU took longer than the previous hard-wired timeout. The Makefile targets for running tests have all been updated to use the new test mechanism. Signed-off-by: Nicko van Someren <[email protected]>
The Makefile builds are now more robust in the face of outdated content. The harness.py code properly ensures that the UEFI boot parameters are correctly set before tests are run on QEMU. The firmware builder scripts have been consolidated into one, parameterised script. All builder support scripts have been tidied into the port's tools directory. Signed-off-by: Nicko van Someren <[email protected]>
Documentation and Makefile have been updated to point to the correct example code location. Signed-off-by: Nicko van Someren <[email protected]>
Summary
This PR introduces a new port to allow MicroPython to run on a PC under the UEFI firmware, prior to a full OS booting. While PCs are not microcontrollers, the pre-boot firmware environment has similarities both in terms of the limited platform and the direct access hardware (since the code runs in "Ring 0"). The port supports both Intel (x64) and ARM (aa64) UEFI platforms.
The UEFI port provides comprehensive wrappers around most of the UEFI API surface. This allows Python code to discover and access hardware, read and write firmware variables and settings, directly interact with extensible EFI protocols and manage the boot process, up to and including implementing custom OS bootloaders in Python.
Most common blocking operations (
stdio,socket,sleepetc) supportasyncioand async support for UEFI events is provided.The port provides access to firmware networking on machines which support this, using the standard MicroPython
networkmodule. Note that this initial port lacks support for Wi-Fi pending hardware and drivers for testing this. TLS support can be enabled either using mbedTLS or by connecting to the EFI_TLS protocol if the host's firmware provides this. The choice is compile-time; using EFI_TLS saves 190KB of 610KB but not all network-enabled firmwares support this.NOTE: The build process for this port makes extensive use of Docker to handle the cross-platform nature of the port.
Testing
This code has extensive UEFI-specific testing that runs under the QEUM PC emulator within Docker containers, as well as supporting QEMU testing in local machines where possible. These tests cover all the parts of the UEFI API surface that I could find a way to test. The PR also includes a set of sample code that exercises the UEFI API which can be used for manual testing.
Trade-offs and Alternatives
This code was designed specifically to make minimal changes to the base MicroPython distribution (and succeeded, since it makes no changes). In practice no compromises needed to be made, which vindicates the MicroPython porting API. The code here supports TLS through either mbedTLS (the default) or use of the EFI_TLS protocol. This is controlled by a compile-time switch (
TLS=mbedtls|efi|none). While the UEFI spec provides a protocol for supporting TLS (a) many platforms don't implement this and (b) when they do it's often based on an outdated version of OpenSSL. On the other hand, using a firmware-provided version of TLS cuts the binary size by about 30%. In most cases the mbedTLS version should be used by if a PC OEM were to want to include MicroPython in their ROM the EFI_TLS version would be more appropriate.Generative AI
Some of the code here was typed by Claude, but the architecture, design and planning was entirely human!