RFC: Do we need (better) support for runtime initialization of C modules

All (?) modules in the MicroPython source code and external C modules are created by defining a const mp_obj_module_t object which basically gets populated at compile time by specifying the globals table. This was done so because it allows putting everything in ROM, I assume. The module object then somehow has to make it into mp_builtin_module_table so it can be found by the import logic. However another way to make modules available within MicroPython is creating an mp_obj_module_t at runtime and register it. Essentially something like

const qstr module_name = qstr_from_str("mymodule");
mp_obj_module_t *module = (mp_obj_module_t*) mp_obj_new_module(module_name);
//Populate the module with variables/functions/classes/....
mp_obj_dict_store(module->globals, "global_var", mp_const_true);
...
//Register module so 'import mymodule' in Python now works
mp_module_register(module_name, module);

this offers some flexibility for making certain things available or not based on runtime conditions for instance. Or if you're short on ROM it allows creating your module in RAM instead (well, just a small part i.e. the globals table not sure if that really helps). It's similar to what gets done in the mpy_init function of native modules, it's also something which is standard in CPython (C modules have a PyInit_xxx which gets called upon first import) and actually there's an example of something similar in unix/main.c (although that just registers a type globally not in a specific module, but the principle is the same).

However there isn't really (see below) any build support which allows doing this: there's no way to say 'here's an initialization function for a module X, call it to create the module X when encountering import X. Is that something we need? I know I could use it because literally all my modules work like that, just not sure if other people ever felt any need for it. Or even thought about it. Implementation could be relatively simple I think: a table similar to how mp_builtin_module_table works but with key/value pairs of module name and a pointer to their init function which gets called to initialize the module once, and a cache for the loaded ones.

Addendum: currently there are some ways to have these 'runtime initialized modules':

you can just patch e.g. main.c and add whatever code you need to create a module
I found a way to leverage the external C module stuff to do it but it's a bit hacky and needs some boilerplate for each module. Idea is that you enable MICROPY_MODULE_BUILTIN_INIT, create a non-const mp_obj_module_t where the globals just contain one __init__ entry, pointing to a function which then replaces the module globals with a RAM dict populated with whatever the module needs. See https://github.com/stinos/micropython-wrap/blob/master/tests/cmodule.c for example

Do I understand correctly that you are talking about built-in C modules, as opposed to dynamically loaded native modules?

C modules yes, but not necessarily built-in (into the core), also the ones using the USER_C_MODULES system

Or if you're short on ROM it allows creating your module in RAM instead (well, just a small part i.e. the globals table not sure if that really helps).

I'm a bit confused by this, because surely whatever you need to create the module in RAM (i.e. code to build up the tables) had to be in ROM anyway. (And code is a less efficient way to represent that than a static const table in ROM).

this offers some flexibility for making certain things available or not based on runtime conditions for instance.

One thing you could do is have two different const (ROM) globals tables and their corresponding dicts (i.e. a base one and one with extra features enabled at runtime), and then you only need to put the mp_obj_module_t in RAM and modify its .globals member in init (via MICROPY_MODULE_BUILTIN_INIT). (This is pretty much what you said right at the end, just using a ROM dict/table rather than populating a RAM dict in code).

To save ROM you could even have one version of the table, with the "extra" stuff at the end, and then define two (also const) mp_obj_dict_t that point to the same table but with different lengths.

whatever you need to create the module in RAM (i.e. code to build up the tables) had to be in ROM anyway

Yes you're right, my mistake: I was thinking in terms of dynamically loaded modules

Continuing some discussion from #5643

Native modules and C modules using USER_C_MODULES aren't interchangeable due to the way the module's globals dict is populated, so that means people have to make a choice between the 2 at one point and going back and forth between the 2 ways isn't super trivial.

I made a few of the extmod modules compile as either static or dynamic using the MICROPY_ENABLE_DYNRUNTIME option. So it is possible to maintain both versions together.

Yes but that was sort of my point: maintaining just one version would be better.

And, if the need is there, we could possible make it so the dynamic module also has a static globals dict (might be worth the effort).

That's also a possibility yes, but I'm personally more leaning towards one unified way for all modules with an initialization function. It just makes more sense to me. And it's compatible with a static globals dict (the initialization function just needs to set the module's globals to point to that dict) whereas the other way around is harder.

why isn't this a thing? That would simplify so much. For instance I want to write my io in python instead of C but I can't do it because I don't want to write whole IO but I can't just write FileIO in python because it is linked at compile time. If there was initialized method for each module that is called when module is first loaded it would be so much better...

From the RP2/Pico world, I maintain a whole suite of C++/C drivers which have individual bindings into MicroPython. Many of these involve a specific, single "base" board (notably not populated with an RP2040 itself) with a variety of features/pins/peripherals which can only be used in a singleton pattern. As such it often makes sense for us to roll a "module" type USER_C_MODULE (we have many drivers implemented as classes to).

The trouble with modules- and I'm not sure this issue specifically addresses this, but it might be tangentially related at least- is that they do not have an initializer/finaliser.

In Python you can be a little bit naughty and include code in your "module.py" to run when it is imported. You can also use "atexit" to register a teardown function. There's no analog for this pattern in MicroPython.

So we end up with singleton modules to drive a specific board which are set up using an "init()" function (called by the user) and then not torn down with a finalizer during soft-reset- so hardware resources are never released. Our solution is to keep a static variable in C that denotes "init already done" and then only set-up the things that soft-reset resets on subsequent "init()"s.

So, my two cents on this is, does it make sense to have:

module initializers
module finalizers

To cover these use-cases, where we want less busywork for the user but have to manage device resources carefully?

The trouble with modules- and I'm not sure this issue specifically addresses this, but it might be tangentially related at least- is that they do not have an initializer/finaliser.

See last paragraph of first post: there is a workaround to get an initializer.

Merge pull request micropython#5654 from pewpew-game/pygamer-pybadge-…

Navigation Menu

Repository navigation

RFC: Do we need (better) support for runtime initialization of C modules #5654

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RFC: Do we need (better) support for runtime initialization of C modules #5654

Description

Activity

tve commented on Feb 17, 2020

stinos commented on Feb 17, 2020

jimmo commented on Feb 18, 2020

stinos commented on Feb 18, 2020

stinos commented on Feb 20, 2020

Enerccio commented on Oct 10, 2021

Gadgetoid commented on Oct 20, 2021

stinos commented on Oct 21, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions