ESP8266 support - #4

@dpgeorge can you take a look at this PR?

It appears like the ESP8266 doesn't support position-independent code, which is currently a requirement for persistent native.

Can you please elaborate on this point? Even if the xtensa compiler doesn't support -fPIC it still seems to generate code that is position independent enough for our purposes. Are you able to give an example of some machine instruction (and corresponding C code) that must be relocated for xtensa?

Yes. As an example, here is modx.elf from modx.c with almost everything commented out, except adding the add1 function using mp_store_global:

Disassembly of section .all:

00000000 <header>:
   0:   4d 02 03 10 06 06 05 00 14 00 00 00 00 00 00 00     M...............
  10:   48 00 00 00                                         H...

00000014 <init>:
  14:   0238            l32i.n  a3, a2, 0
  16:   f0c112          addi    a1, a1, -16
  19:   026d            mov.n   a6, a2
  1b:   1228            l32i.n  a2, a2, 4
  1d:   21c9            s32i.n  a12, a1, 8
  1f:   2c2372          l32i    a7, a3, 176
  22:   93c8            l32i.n  a12, a3, 36
  24:   fffb51          l32r    a5, 10 <header+0x10>
  27:   130c            movi.n  a3, 1
  29:   3109            s32i.n  a0, a1, 12
  2b:   11d9            s32i.n  a13, a1, 4
  2d:   034d            mov.n   a4, a3
  2f:   32d8            l32i.n  a13, a2, 12
  31:   020c            movi.n  a2, 0
  33:   0007c0          callx0  a7
  36:   023d            mov.n   a3, a2
  38:   0d2d            mov.n   a2, a13
  3a:   000cc0          callx0  a12
  3d:   3108            l32i.n  a0, a1, 12
  3f:   21c8            l32i.n  a12, a1, 8
  41:   11d8            l32i.n  a13, a1, 4
  43:   10c112          addi    a1, a1, 16
  46:   f00d            ret.n

00000048 <modx_add1>:
  48:   0228            l32i.n  a2, a2, 0
  4a:   f0c112          addi    a1, a1, -16
  4d:   e258            l32i.n  a5, a2, 56
  4f:   3109            s32i.n  a0, a1, 12
  51:   c21c            movi.n  a2, 28
  53:   340c            movi.n  a4, 3
  55:   0005c0          callx0  a5
  58:   3108            l32i.n  a0, a1, 12
  5a:   10c112          addi    a1, a1, 16
  5d:   f00d            ret.n
        ...

The first callx0 instruction executes mp_obj_new_fun_extern and the second executes mp_store_global. We mostly care about mp_obj_new_fun_extern, as it takes the pointer to the modx_add1 function.

This pointer is loaded in the L32R instruction at 0x24 into register a5, which is the fourth argument according to the Xtensa calling convention (documentation, see 8.1.4 Argument Passing). And indeed, the function pointer is the fourth argument of mp_obj_new_fun_extern.

Translated to ARM assembly this L32R line would be something like ldr r5, [pc, #-something]. It does a PC-relative load of the function pointer. Being a PC-relative load it works just fine. But you can see which address it loads at 0x10, which is 0x000048. This address is treated as an absolute address, and it matches the address of modx_add1 in the disassembly. When this function is called using callx0 the program will crash as it tries to execute the absolute address 0x48.

Note that there is also a relocation emitted, the important bits are:

 Offset     Info    Type            Sym.Value  Sym. Name + Addend
00000010  00000101 R_XTENSA_32       00000000   .all + 0

There is almost no documentation about the various relocations on the internet, but from what I found this essentially means "take the address of the target and add the program counter", which of course is only known at runtime.

I've tried all possible options (-fpic, -fPIC, -fpie, -fPIE, no option) and they all produced the exact same .o and .elf file (the same md5sum). So it appears like support is non-existent in gcc. I even looked through the ISA documentation and couldn't find a reference to a program counter register. It appears like the Xtensa ISA itself doesn't support position-independent execution at all.

Summary: I cannot find a way to avoid runtime relocations with the given C code.

Thanks for the detailed reply. I now understand the problem a bit better.

As a simple test I tried compiling the esp8266 port with "-fPIC" and it did produce different code... but it still used callx0 in some places, which (as you point out) does an absolute call.

Did you try compiling without the -mlongcalls option? That option forces callx0 so it can reach code beyond the call0 limit of +/-500k, but without this option the compiler generates call0 for most (if not all) calls.

The code is already compiled without -mlongcalls. Adding -mloncalls or -mno-longcalls does not change the resulting binary (in my test).

The problem is that mp_obj_new_fun_extern takes a function pointer to create a Python function object. In the end the function pointer has to have an absolute address, somehow.

It may be possible to somehow add the code base address manually to the relative function pointer (e.g. within mp_obj_new_fun_extern). This should work, but feels like a hack. When I tried it, I didn't get it working though.

The problem is that mp_obj_new_fun_extern takes a function pointer to create a Python function object. In the end the function pointer has to have an absolute address, somehow.

Right. On archs that do support -fPIC this absolute address is computed using PC-relative instructions (or similar) which means that the location of the code can change from run to run.

It may be possible to somehow add the code base address manually to the relative function pointer

This would be like implementing a poor-mans -fPIC in the actual C code, using lots of macros/tricks.

If we can't rely on -fPIC working for any given arch then it looks like we want to go down the path of allowing relocations in native code, and linking when the native .mpy file is loaded. One of the benefits of doing it this way is that the native code is smaller because PIC adds overhead.

Rebased the commits on dpgeorge:dynamic-native-modules-v2.

This would be like implementing a poor-mans -fPIC in the actual C code, using lots of macros/tricks.

Yeah, I'm afraid there is no other way.
And something which I hadn't thought about before: pointers to constants (e.g. arrays) are also affected by position independent code, for similar reasons. This is probably much more common than function pointers, and are also used in the init() of modx (to generate qstrs from char arrays). Such a -fPIC workaround would involve macros around all those memory accesses, so that the C module code doesn't really look like C anymore.

One of the benefits of doing it this way is that the native code is smaller because PIC adds overhead.

True, but the overhead isn't as big as you might think. I've extended relocation support to Thumb and the result is a modx.mpy file size reduction of 8 bytes (329 → 321). Without relocation overhead, the loaded machine code (often in RAM) is 16 bytes smaller. At the same time, the firmware size increases by 36 bytes (tested on a Cortex-M0). I've put the (unfinished) result here: aykevl@db79abd

I think there are other more interesting benefits of relocations, as I (and you) have discussed before.

I think there are other more interesting benefits of relocations, as I (and you) have discussed before.

Yes. So maybe it's time to make a list (based on those discussions) of pros/cons for relocation vs "PIC". Then we can make a decision and move to implement it.

PIC vs relocations, as I see it.

Advantages of PIC:

• .mpy files can be directly executed from flash.
• Firmware size is a bit smaller (no relocation code)

Advantages of relocations:

• Some architectures do not support PIC, in particular Xtensa (and probably PIC16 from what I've read). It would require a lot of workarounds to make the code position-independent.
• It becomes possible to eliminate the context variable. This would lead to cleaner C code without loads of macros, probably greatly reduced .mpy file size, and likely increased execution speed. I haven't tested this one on this branch but it should be possible (as I did something similar before).
• Module members (stuff like modx.add1) can be stored statically in the .text of the .mpy file. It may even be possible to remove the init function entirely so that loading .mpy modules becomes more like how built-in C modules (like uos) are loaded, with module globals stored in a special struct at the start.
• .mpy file size is a little bit smaller without -fPIC due to PIC-overhead.
• Firmware size might actually be reduced because mp_obj_new_fun_extern might be removed.
• Non-const global data (.data) can be used more easily, when .text and .data are stored in a separate memory region (e.g. storing .text in flash and .data in RAM, see Compile Python Scripts On the Go micropython/micropython#2709).

Regarding the first bullet point: so far I haven't seen a concrete use case for direct execution from flash (without loading it first). Either the module is stored on a filesystem (which does not allow direct execution due to fragmentation) or the module is built in the firmware, in which case there is no reason to store it as .mpy file instead of a built in C module (like uos, btree, etc.).

It may also be possible to only use relocations on Xtensa, keeping PIC code on all other architectures. This avoids the PIC workarounds but does not give all other advantages.

In the end I did go with relocations, although only in a simple form in that there is a single GOT which is relocated just by adding a constant (pointer to the start of the .text section when loaded) to each entry.

See the result at micropython#5083 . That implements a full linker to turn .o into .mpy and supports x86, x64, ARM Thumb and xtensa (esp8266 and esp32).