Repository navigation
MQTT to AWS IoT Core fails with mbedtls_ssl_handshake_error #5929
Description
Activity
mbedtls_ssl_handshake error: -2880
This is
MBEDTLS_ERR_X509_ALLOC_FAILED, so it's running out of RTOS heap memory.Nope, it's the C heap, not the MP heap... Either use a module with PSRAM or reduce mp_task_heap_size by some factor (e.g. 80%) here https://github.com/micropython/micropython/blob/master/ports/esp32/main.c#L103-L104 and rebuild the fw and try that...
uriously I tried the same code that worked on earlier firmware (20191220) doesn't work on the latest one 20200416
That's most likely due to an update to the ESP-IDF, which changes the heap layout and has less RTOS heap available.
Hi @Aish-Git see the long explanation in #5543
Thanks @Carglglz - incredibly helpful and sorry didn't see that before I logged this one. Let me try some options listed there in terms of a 1096 bit key to keep the cert length small. I am also going to try the recommendation that @tve provided on increasing the C heap size. Will keep this thread posted...
I also received the ssl_handshake_status error. My error log is as the following
ssl_handshake_status: -256 Cannot connect MQTT: [Errno 5] EIO Traceback (most recent call last): File "main.py", line 236, in <module> File "main.py", line 217, in connect_mqtt File "umqtt/simple.py", line 62, in connect OSError: [Errno 5] EIOI'm using AWS IoT with umqtt. Sometimes it get connected but sometime it gives me the above error. Can anyone help me with this?
I make SSL connections and use AmazonIOT and my memory configuration is this.
`size_t mp_task_heap_size = 50 * 1024; //heap_caps_get_largest_free_block(MALLOC_CAP_8BIT);void *mp_task_heap = malloc(mp_task_heap_size);`
Downgrading to 1.12 (esp32-idf3-20191220-v1.12.bin) fixed it for me. 1.13 must do something different.
@eliclement I never did get to going down that path as I switched over to a ESP Pico board (TinyPico) that has an onboard SRAM so ended up working well for me.
@Aish-Git OK thanks. I had to remove the BLE in the build and which freed some heap memory and it was sufficient.
I had similar issues when connecting to RabbitMQ (actually I think that AWS IoT is using RabbitMQ under the hood). I am not sure if you have control over the Ciphers. For me, I controlled it server side and using DHE-RSA-AES128-GCM-SHA256 i worked without any changes to heap. But anyway, more RAM would always be great =) .
Is it possible to restrict the Ciphers Mbedtls is using? Then, I think AWS IoT will also allow using DHE-RSA-AES128-GCM-SHA256 because it is still recommended and you can connect without issues.
Downgrading to 1.12 (esp32-idf3-20191220-v1.12.bin) fixed it for me. 1.13 must do something different.
Thank you for that, @srcnet2 . I've been working on this for a while and v1.12 idf3 binary is the only one that's able to reliably connect to AWS using a generic ESP32.
AWS for IoT is the 800 pound gorilla. Sure would be nice to use a generic ESP32 with the latest binary in an AWS application. Using idf4, "esp32-idf4-20210202-v1.14.bin" results in:
OSError: (-17168, 'MBEDTLS_ERR_RSA_PRIVATE_FAILED+MBEDTLS_ERR_MPI_ALLOC_FAILED')@sactre said...
I make SSL connections and use AmazonIOT and my memory configuration is this.
`size_t mp_task_heap_size = 50 * 1024; //heap_caps_get_largest_free_block(MALLOC_CAP_8BIT);void *mp_task_heap = malloc(mp_task_heap_size);`
If that's all it takes, it might be of value to make available updated binaries that are capable of connecting to AWS IoT Core using a generic ESP32.
I have good news and I have bad news! The new version of MicroPython, idf4 v1.15 -- esp32-20210418-v1.15.bin, works with AWS IoT! YAY!!!
v1.15 does not have mqtt built in so you have to install your own. That's OK because it's easy to do.
Now for the bad news. If I drop into REPL a few times while running my tests, eventually I get a WiFi error that looks like this:
Traceback (most recent call last): File "<stdin>", line 23, in <module> OSError: Wifi Internal ErrorI don't know why, and the message doesn't give me much to go on. I have to machine.reset() to get the WiFi to work again. A soft boot (^D) doesn't do it.
I will open another issue about this specific WiFi problem.
I can also force my WiFi station mode inactive in my case station.active(False), and that works to reset the WiFi, but then my software has to reconnect every time when coming back from REPL. Is it supposed to work like this?
My solution was to make Micropython leave some of the heap for the RTOS, instead of claiming all of free memory for itself. Experimentation showed that
- 2K reserved never works
- 4K reserved sometimes works
- 8K reserved always works
But this probably depends on what other features are available.
diff --git a/ports/esp32/main.c b/ports/esp32/main.c index ff6dd6957..0149e8a0b 100644 --- a/ports/esp32/main.c +++ b/ports/esp32/main.c @@ -121,6 +121,7 @@ void mp_task(void *pvParameter) { #else // Allocate the uPy heap using malloc and get the largest available region size_t mp_task_heap_size = heap_caps_get_largest_free_block(MALLOC_CAP_8BIT); + mp_task_heap_size -= 8192; // Reserve 8K for C heap (mbedTLS needs it) void *mp_task_heap = malloc(mp_task_heap_size); #endifDoes it make sense to make this a configuration option?
My board has no SPIRAM, so I have no idea if this is necessary if there is plenty of RAM.
By default mbedtls only uses internal ram on the esp32 (security concerns). However esp-idf provides a config to allow mbedtls to alloc also in the PSRAM space:
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/kconfig.html#config-mbedtls-mem-alloc-modeSolution proposed by @ironss above did the trick for me. After recompiling the latest trunk version of micropython (October 13th 2022), I can now connect to AWS IoT and send and receive messages.
Big thank you @ironss, spent about 3 days before I stumbled upon this solution.
EDIT: I compiled micropython against esp-idf 4.3.4 (latest I found)
How would we go about debugging
MBEDTLS_ERR_SSL_BAD_HS_CLIENT_HELLOerrors? I have attached my code for reference... on ESP32-C3. I'm using the latest nightly of micropython for esp32-c3 (esp32c3-usb-20230411-unstable-v1.19.1-1014-gbde222ce8.bin), the latest version of mqqt_asimport gc import time import machine import network import json gc.collect() import ussl as ssl import uasyncio as asyncio from ubinascii import hexlify gc.collect() from net import connect # just a wrapper for Wifi connection from lib.mqtt_as import config from lib.mqtt_as import MQTTClient gc.collect() client_id = hexlify(machine.unique_id()).decode('utf-8') # Set up the AWS IoT endpoint details endpoint = "xxxxxxxxxxxxx-yyy.iot.us-west-2.amazonaws.com" topic = "sdk/test/python" # Set up the AWS IoT certificate and key files # cert_file = "/flash/cert/esp32-c3.cert.pem" cert_file = "/flash/cert/device.cert.der" # key_file = "/flash/cert/esp32-c3.private.key" key_file = "/flash/cert/device.private.der" # root_ca_file = "/flash/cert/root-CA.crt" # Path to the root CA certificate file root_ca_file = "/flash/cert/root-CA.der" # Path to the root CA certificate file PVT_KEY = open(key_file,'rb').read() CERT_KEY = open(cert_file, 'rb').read() CA_KEY = open(root_ca_file, 'rb').read() ssl_params = { 'key': PVT_KEY, 'cert': CA_KEY, 'server_side': True, 'server_hostname': endpoint, 'do_handshake': True, 'cert_reqs': ssl.CERT_REQUIRED, 'cadata': CA_KEY } # Connect to AWS IoT MQTT def connectMQTT(): internet_connection = connect() print("MQTT CONNECTING", internet_connection) config['client_id'] = client_id config['server'] = endpoint config['port'] = 8883 config['keepalive'] = 5000 config['ssl'] = True config['ssl_params'] = ssl_params config['ssid'] = internet_connection['ssid'] config['wifi_pw'] = internet_connection['password'] MQTTClient.DEBUG = True client = MQTTClient(config) try: asyncio.run(client.connect(quick=True)) print("MQTT CONNECTED") #, client.isconnected()) finally: print("MQTT CLOSED") #, client.isconnected()) # client.close() # client.disconnect() return client # Publish a message to the topic def sendMQTT(client, message): print(f"Sending: {client.isconnected()} {message}") client.publish(topic, message) connectMQTT()Tried the stable micropython as well, doesn't seem to make any difference.
Connecting to broker. Traceback (most recent call last): File "<stdin>", line 9, in <module> File "util.py", line 87, in <module> File "util.py", line 70, in connectMQTT File "uasyncio/core.py", line 1, in run File "uasyncio/core.py", line 1, in run_until_complete File "uasyncio/core.py", line 1, in run_until_complete File "lib/mqtt_as.py", line 637, in connect File "lib/mqtt_as.py", line 271, in _connect File "lib/ssl.py", line 30, in wrap_socket OSError: (-30976, 'MBEDTLS_ERR_SSL_BAD_HS_CLIENT_HELLO')Hello, @ironss. How would you go about doing this on micropython. Should I edit the binary files or something? Its still 2023 and none of the solutions work so far. I am trying out different versions to test stuff.
I can think of four options.
- Use an ESP32 board that has SPIRAM. The uPy heap uses the SPIRAM, leaving the internal RAM for the C heap, for TLS, BT, file system file descriptors, etc. My project needed about 20 BLE GATT services, and about 100 GATT characteristics, which also use C heap. I eventually went for this option because if I made the C heap big enough for BLE, TLS, etc, there was not enough internal RAM left for the application to use.
- Download the source code, edit it as described, then compile your own build and install it on your board. The instructions for ESP32 are detailed, and Just Work if you follow them accurately. I had to do this too, to increase the BLE settings (and to solve the original TLS problem).
- Edit a binary image of uPy on a PC and install the edited version on your board. Good luck, you're on your own.
- Write a uPy script to edit the flash directly on the board. You're even more out on a limb.
- sgi
mbedtls_ssl_handshake error: -2880
I (14912) wifi:Traceback (most recent call last):
File "main.py", line 99, in
File "main.py", line 96, in
File "main.py", line 69, in mqtt
File "umqtt/simple.py", line 61, in connect
OSError: [Errno 5] EIO
Fails on the latest firmware but works on any of the 2019 builds - tried it with esp32-idf3-20191220-v1.12.bin and it works but needed BLE and hence upgraded to 2020 versions and fails on all the March/April releases.