py/modsys.c: Add sys._exc_traceback. - #11244

Code size report:

Reference:  docs/library/network.PPP: Document the optional poll() argument. [115f413]
Comparison: py/objexcept: Add __traceback__ attribute getter. [merge of 22bb986]
  mpy-cross:  +240 +0.064% 
   bare-arm:  +104 +0.184% 
minimal x86:  +268 +0.143% 
   unix x64:  +280 +0.033% standard
      stm32:  +104 +0.026% PYBV10
      esp32:  +140 +0.008% ESP32_GENERIC
     mimxrt:  +104 +0.028% TEENSY40
        rp2:   +96 +0.010% RPI_PICO_W
       samd:  +104 +0.038% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:  +116 +0.025% VIRT_RV32

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.41%. Comparing base (1095bbb) to head (22bb986).
⚠️ Report is 6 commits behind head on master.

@@           Coverage Diff           @@
##           master   #11244   +/-   ##
=======================================
  Coverage   98.41%   98.41%           
=======================================
  Files         174      174           
  Lines       22325    22338   +13     
=======================================
+ Hits        21972    21985   +13     
  Misses        353      353           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

First thing to ask is how would one do this in CPython?

Then we need to consider if it's worth the cost (code size) and whether it should be guarded by a config macro. I would say yes it should be, but then at what level would it be enabled, eg MICROPY_CONFIG_ROM_LEVEL_EXTRA_FEATURES.

You can get this info in CPython is like this:

import traceback

def foo():
    1/0

try:
    foo()
except Exception as e:
    tb = e.__traceback__
    while tb is not None:
        print("====")
        print("  filename: " + str(tb.tb_frame.f_code.co_filename))
        print("  line: " + str(tb.tb_lineno))
        print("  name: " + str(tb.tb_frame.f_code.co_name))
        tb = tb.tb_next

This involves at least three different classes: traceback, frame, and code. The latter two contain quite a few things:

dir(tb): ['tb_frame', 'tb_lasti', 'tb_lineno', 'tb_next']
dir(frame): ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'clear', 'f_back', 'f_builtins', 'f_code', 'f_globals', 'f_lasti', 'f_lineno', 'f_locals', 'f_trace', 'f_trace_lines', 'f_trace_opcodes']
dir(code): ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_kwonlyargcount', 'co_lines', 'co_linetable', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_posonlyargcount', 'co_stacksize', 'co_varnames', 'replace']

The Python standard library also comes with a traceback module that introduces more classes. MicroPython implements a minimal traceback module.

CPython does not have sys.print_exception like MicroPython does.

I changed it so that this function is only enabled by default if MICROPY_CONFIG_ROM_LEVEL_EXTRA_FEATURES is 1. (I'm glad to see this provides it for the RP2 boards.) I also added documentation.

This is an automated heads-up that we've just merged a Pull Request
that removes the STATIC macro from MicroPython's C API.

See #13763

A search suggests this PR might apply the STATIC macro to some C code. If it
does, then next time you rebase the PR (or merge from master) then you should
please replace all the STATIC keywords with static.

Although this is an automated message, feel free to @-reply to me directly if
you have any questions about this.

Thanks for keeping this PR alive. I can see how it would be useful.

To reduce code size I suggest implementing the functionality as an attribute of the exception type, rather than as a new function. So it would go in mp_obj_exception_attr(). To keep code size further down, it could reuse the existing __traceback__ attribute name. So the example from above would be:

import sys

def foo():
  1/0

try:
  foo()
except Exception as e:
  sys.print_exception(e)
  print(e.__traceback__)

Now, that's not fully compatible with CPython, but it's a little bit closer than having a separate sys._exc_traceback() function. We can say that if you want compatibility with CPython then you must consider e.__traceback__ as an opaque (or implementation defined) value, and then use the traceback module to convert that opaque value to something usable. Eg traceback.extract_tb(e.__traceback__) would work the same in CPython and MicroPython, even though the underlying value of e.__traceback__ was different.

And then if you don't care about CPython compatibility and you want to efficiently get the traceback data, you can use the value of e.__traceback__ directly (although maybe we should document it as opaque and subject to change).

I think this is ready to be merged in. I changed it to make a list of tuples, so we can easily extend it in the future by adding to the tuple, and well-written user code will not break. And there are tests and documentation now.

Perhaps the "minimal" build on Unix should not have this feature because it doesn't track line numbers.

This increases the size of minimal ports by quite a bit. But I think it's useful to have, and IMO more useful than sys.print_exception() which is enabled on all builds. So I propose that we disable sys.print_exception() on minimal builds in favour of exception.__traceback__. That should claw back a lot of the code size change.

But, let's do that in a separate PR, because it'll probably need changes to the test suite.

@DavidEGrayson are you able to respond to the above review? If not then I can make my suggested changes during rebase and merge of this PR.

@DavidEGrayson I'll leave this up to you, let me know after you've had a chance to get back to it.