Repository navigation
RFC: change ussl to implement SSLContext #8915
Description
Activity
1. Should probably keep the existing `ussl.wrap_socket()` for backwards compatibility for a couple of releases.Yes it should stay for a while.
The question is if we should add, as a MicroPython-extension, a way to pass in data directly
I don't think this was ever needed because key and cert are read from files anyway before being passed as buffers, and I just don't see the reasoning behind reading/extracting key/cert from some external source, because that's just as (in)secure as storing them in files (so may as well just store them in files). That said, these kwargs should still be supported for backwards compatibility with code that uses
wrap_socket, but if possibleload_cert_chainshould fail if it's used with these kwargs. I'm not sure how to implement that, but this way whenwrap_socketis gone these kwargs can go as well3. Whether to implement it in C or Python.I'm not sure if it matters, but at some point I want to support URIs for key/cert in crypto devices, so may make sense to keep things in C for now if it makes it easier to support that.
I have encountered issues where failing to explicitly ".close()" a wrapped socket immediately after use will eventually lead to system level OOM errors, and that's with - imho - fairly trivial programs. With many of our C drivers using small amounts of RAM here and there it has become quite dicey!
Would this potentially allow returning some of the large (25K?) chunk of RAM taken from MicroPython to the gc_heap?
Excellent! That's will be a amazing feature! :)
I have encountered issues where failing to explicitly ".close()" a wrapped socket immediately after use will eventually lead to system level OOM errors
You must always close sockets to free up internal resources. lwIP has a memory pool independent to the uPy heap, and the only way to free lwIP memory is by a socket close. Similar with SSL wrapped sockets, you must close them to free up memory.
They shouldn't need to be close immediately, but definitely once they are no longer needed (and before you create a new one).
Would this potentially allow returning some of the large (25K?) chunk of RAM taken from MicroPython to the gc_heap?
Do you mean RAM taken when you create a wrapped socket? If you close the socket that RAM should be returned to the uPy heap automatically. (Note that on esp32 wrapped sockets take RAM from the FreeRTOS heap, not the uPy heap.)
Do you mean RAM taken when you create a wrapped socket? If you close the socket that RAM should be returned to the uPy heap automatically. (Note that on esp32 wrapped sockets take RAM from the FreeRTOS heap, not the uPy heap.)
Will this RFC also apply to esp32 or won't it be able to, due to mentioned differences?
Currently we're forced to make sure that wrapping a socket is one of the first things we do, (before enabling BLE i.e.).
Otherwise there's the risk of ENOMEM errors. it would definitely be nice to have more control over memory usage on the ESP32 as well.Reacted by beyonloWill this RFC also apply to esp32 or won't it be able to, due to mentioned differences?
Currently we're forced to make sure that wrapping a socket is one of the first things we doThe idea is to be able to do this:
import ssl # pre-allocate context and SSL buffers ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) # do lots of other stuff ... # use ctx, uses pre-allocated SSL buffers s = ctx.wrap_socket(...) s.close() # reuse ctx, reuse buffers s = ctx.wrap_socket(...) s.close()
At least, that's the idea. And the API is still compatible with CPython. Not sure how this will actually be implemented on esp32 though.
Note that on esp32 wrapped sockets take RAM from the FreeRTOS heap, not the uPy heap.
Not sure how this will actually be implemented on esp32 though.@dpgeorge I now have ESP32 working with mbedtls using the uPy heap via m_tracked_alloc (and Rob's split heap PR for more heap). So I don't think ESP32 will be any different to the other ports in this regard. (Also, FWIW, https requests that previously failed now work on IDF 4.4 with this).
Reacted by Chris Overgaauw, Carglglz and Adam ChasenDo you mean RAM taken when you create a wrapped socket? If you close the socket that RAM should be returned to the uPy heap automatically. (Note that on esp32 wrapped sockets take RAM from the FreeRTOS heap, not the uPy heap.)
I meant specifically this change -
which dramatically reduces the amount of RAM available to MicroPython... quite painful when you're trying to juggle network sockets and screens 😬Lines 56 to 62 in 15fea3a
But I fear the answer is no if the
lwIPmemory pool remains separate.I now have ESP32 working with mbedtls using the uPy heap via m_tracked_alloc
Fantastic!! That will make a big difference for esp32 and SSL.
which dramatically reduces the amount of RAM available to MicroPython [on rp2]
Right, so lwIP has statically allocated memory pools, defined by
ports/rp2/lwip_inc/lwipopts.h. It's currently tuned for reasonable TCP performance without taking up too much RAM. But you can easily reconfigure it to use less RAM, at the expense of throughput. Seeports/stm32/lwip_inc/lwipopts.hwhich has 4 levels of RAM usage.Apart from that, I think we can actually increase the 166*1024 value. Let me check.
Thank you @dpgeorge I may have to look into this.
Someone off-topic, but the sad truth is that we're also making quite a lot of unchecked, temporary (and not so temporary) allocations outside of gc_heap in our C++ drivers, so the extra memory pressure in RP2 + W has caused some things that (albeit through pure blind luck) worked before to hardlock. Any savings to RAM usage (or even any additional knowledge about how it works) would buy me some time while I try to fix these up using the new tools (tracked alloc, more flexible root pointers) you've provided. As such I'm keeping an especially close eye on the project lately.
allocations outside of gc_heap in our C++ drivers
Where is this memory coming from, is it from the uPy gc_heap, or elsewhere in RAM?
I tried increasing the MICROPY_GC_HEAP_SIZE to 180*1024 and that compiles, and leaves 11k of C stack. Compare that to 32k stack for the Pico and 25k stack for standard Pico-W. 11k should be enough, and there are runtime checks to make sure it doesn't overflow. I suggest trying this value (180k heap).
Also looks like there is 2k of unused C heap which can be reclaimed by removing the
.heapsection inmemmap_mp.ld.Should probably keep the existing ussl.wrap_socket() for backwards compatibility for a couple of releases. It could print a deprecation warning.
I agree with this.
Whether to implement it in C or Python.
I think it is easier to implement it in Python and extend some features in C, I made demo in #8968 about this, @dpgeorge if you want to check it out 👍🏼
Hi George. I am using this new ssl module (in the latest Micropython version for a raspberry pico W). The reason for me is only better memorymanagement: I often encountered ENOMEM errors during urequests.post calls due to fragmentation.
See your first post (july 2022):
One other benefit of using SSLContext is that it can be used to preallocate the large TLS buffers (16K) needed for a wrapped socket, and the buffer can be reused over and over as long as only one wrapped socket is needed at a time. This will help to reduce memory fragmentation and guarantee that sockets can be wrapped without running out of memory.
So I declare a global varable
a= ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) in the hope that that covers the data for my post requests.
I use this variable "a" later in my request call: s = a.wrap_socket(s, server_hostname=host).
This all works fine but I have doubts about the memory usage: with the help of micropython.meminfo(1) I see that the declaration of "a" takes only 2k memory. Am I doing something wrong here?
Currently MicroPython supports SSL/TLS connections using
ussl.wrap_socket(). CPython long ago replaced this function withssl.SSLContext:And in CPython
ssl.wrap_socket()is deprecated since CPython version 3.7.I propose that MicroPython also switch to using
SSLContext. The reasons to change are:SSLContextprovide more control/options).uasynciocan support TLS clients and servers (CPythonasynciorequires passing in anSSLContextobject toasyncio.start_server()andasyncio.open_connection()).server_hostnamein a CPython-compatible way.One other benefit of using
SSLContextis that it can be used to preallocate the large TLS buffers (16K) needed for a wrapped socket, and the buffer can be reused over and over as long as only one wrapped socket is needed at a time. This will help to reduce memory fragmentation and guarantee that sockets can be wrapped without running out of memory.The proposed
usslmodule would now look like this:Things to discuss/decide:
ussl.wrap_socket()for backwards compatibility for a couple of releases. It could print a deprecation warning.load_cert_chain()takes filenames as arguments and loads the key/cert from a file. This is different to the existingkey/certargs inussl.wrap_socket()(which are not CPython compatible) that take a str/bytes object containing the key/cert data. The question is if we should add, as a MicroPython-extension, a way to pass in data directly like this, and if yes how that would be done. For example add keyword-only args likeload_cert_chain(self, certfile, keyfile, *, keydata, certdata), or a new method likeload_cert_chain_data(self, keydata, certdata).