ESP32: can't access esp-idf from dynamic native module #5458

Activity

1. tve commented on Dec 26, 2019

   

   tve

   on Dec 26, 2019

   ContributorAuthor

   More actions

   I'm seeing that what I'm asking for is not currently possible because the dynamically loaded modules are not really linked against the full firmware image. I.e., the only externs that can be called are the ones in mp_fun_table. For dev purposes it might be neat if one could pull additional symbols from a specific ELF file and add to mp_fun_table, but in general that seems like a can of worms...
   Do I understand all this correctly?
2. dpgeorge

   added

   port-esp32

   on Dec 27, 2019
3. dpgeorge commented on Dec 27, 2019

   

   dpgeorge

   on Dec 27, 2019

   Member

   More actions

   NB: it's odd that one has to use ARCH=xtensawin for the esp32... what does "win" mean?

   It means that the system ABI calling convention uses windowed registers.

   For dev purposes it might be neat if one could pull additional symbols from a specific ELF file and add to mp_fun_table, but in general that seems like a can of worms...
   Do I understand all this correctly?

   That's correct. It shouldn't actually be that hard to add more symbols, eg from the IDF. Just add the ones you need to the end of mp_fun_table (just as void*'s would be fine), then also add them in tools/mpy_ld.py's fun_table dict in the same location. Then mpy_ld.py should be able to pick these symbols up and provide relocations for them when the mpy is imported.
4. tve

   added a commit that references this issue on Dec 27, 2019

   Doc how to add new symbols to mp_fun_table for native modules

   ...

   cb65626
5. tve commented on Dec 27, 2019

   

   tve

   on Dec 27, 2019

   ContributorAuthor

   More actions

   I submitted a tiny PR #5463 to add a paragraph to the docs. Closing this issue with that...
6. tve

   closed this as completedon Dec 27, 2019
7. dpgeorge commented on Dec 28, 2019

   

   dpgeorge

   on Dec 28, 2019

   Member

   More actions

   If this ends up being a frequently requested feature, it might be a good idea to facilitate it by easily allowing port-specific extensions to the function linking table.
8. deshipu commented on Feb 1, 2020

   

   deshipu

   on Feb 1, 2020

   Last edited by deshipu

   Contributor

   More actions

   Sorry for commenting on a closed bug, but I'm having problems following its resolution. I'm trying to include mp_obj_get_array from py/obj.c in the fun_table. So I changed nativeglue.c like this:

   ...
       &mp_stream_unbuffered_readline_obj,
       &mp_stream_write_obj,
       mp_obj_get_array,
   };
   
   #endif // MICROPY_EMIT_NATIVE
   

   and the nativeglue.h like this:

   ...
       const mp_obj_fun_builtin_var_t *stream_unbuffered_readline_obj;
       const mp_obj_fun_builtin_var_t *stream_write_obj;
       void (*mp_obj_get_array)(mp_obj_t o, size_t *len, mp_obj_t **items);
   } mp_fun_table_t;
   
   extern const mp_fun_table_t mp_fun_table;
   
   #endif // MICROPY_INCLUDED_PY_NATIVEGLUE_H
   

   Then I edited tools/mpy_ld.py like this:

   ...
               'mp_stream_unbuffered_readline_obj',
               'mp_stream_write_obj',
               'mp_obj_get_array',
           ])
       }
       for sym in env.unresolved_syms:
           assert sym['st_value'] == 0
   ...
   

   and still I can't get code that uses mp_obj_get_array to compile:

   Traceback (most recent call last):
     File "../../../micropython/micropython/tools/mpy_ld.py", line 965, in <module>
       main()
     File "../../../micropython/micropython/tools/mpy_ld.py", line 962, in main
       do_link(args)
     File "../../../micropython/micropython/tools/mpy_ld.py", line 939, in do_link
       link_objects(env, len(native_qstr_vals), len(native_qstr_objs))
     File "../../../micropython/micropython/tools/mpy_ld.py", line 720, in link_objects
       populate_got(env)
     File "../../../micropython/micropython/tools/mpy_ld.py", line 385, in populate_got
       assert 0, (got_entry.name, got_entry.sec_name)
   AssertionError: ('mp_obj_get_array', '.external.mp_fun_table')
   ../../../micropython/micropython/py/dynruntime.mk:138: recipe for target 'build/_stage.native.mpy' failed
   

   Any advice would be appreciated.
9. dpgeorge commented on Feb 1, 2020

   

   dpgeorge

   on Feb 1, 2020

   Member

   More actions

   Sorry for commenting on a closed bug, but I'm having problems following its resolution.

   @deshipu no problem, it can be a tricky feature to use. I think the problem here is that the custom linker doesn't know how to relocate calls to entries in the function table. Instead you can try just referencing the function in your code using mp_fun_table.mp_obj_get_array(...), like it's done with the other macros in py/dynruntime.h
10. deshipu commented on Feb 1, 2020

    

    deshipu

    on Feb 1, 2020

    Contributor

    More actions

    That worked, thank you!
11. dpgeorge commented on Feb 2, 2020

    

    dpgeorge

    on Feb 2, 2020

    Member

    More actions

    Just to follow up on that point: the linker cannot currently deal with directly calling functions listed in mp_fun_table because the compiler inserts normal calls to such functions and the dynamic loader that runs on the target device cannot relocate such calls (in part because the destination may not be in range of the call machine instruction). The way around this is for the (offline) linker to automatically generate stubs for such functions and include them in the final .mpy. Eg:

    // this is an automatically generated stub
    void mp_obj_get_array(...) {
        mp_fun_table.mp_obj_get_array(...);
    }
    

    This is part of the dynamically loadable native code, and so will be in range of a normal call (and the normal call will be relocated statically to point to the stub).