Repository navigation
Uploading to esp32-s3 (4MB) corrupts file #17865
Description
Activity
How should I read the file dump above? Are the green lines those in which the files differ? If yes, the a file corruption is highly unlikely. It then looks as if you compare different versions of the files, or a file was never written, or to different places..
Sorry for not having been clear enough, @robert-hh!
The (green) lines are the output of diff-ing the original file (as it was uploaded to the device) to the file as it was retrieved (downloaded) from it.
In other words:
mpremote cpcreated a file on the device that had extra lines (the ones in green).Hope this makes sense!
That is a strange phenomenon. I tried to replicate your test with a S3 board, but could not.
Does the upload or download of the file cause the corruption? You mentioned syntax errors, but you can as well just read the file and print the content in REPL to be sure.I use mpremote very often each day to copy files between board and PC, and never noticed any problem. So it may be something not related to mpremote. Sometimes I see echoes in the USB stack, which should not be there. That could cause such text duplication like you see. But that is hard to trace.
Thank you for taking the time @robert-hh.
It appears to occur during the upload, since printing the source code in REPL already has those changes.For what it's worth: I am seeing this on an ESP32-S3-Zero board from Waveshare. I installed the most recent snapshot, since the released 1.25 version wasn't compatible with 4 MB generic targets (as per #17567). I should be able to see if I can verify this with another S3-Zero early next week, if that'd help.
I updated a ESP32 S2 board to v1.26 this morning, and saw exactly the same issue as I described (with an S3 board). Since this didn't happen to with v1.25 on that device, this could be a regression between the two versions.
That's bad. I looked through the changes between v1.25 to v1.26 and could not spot anything that could cause the change, except maybe the version changes of the ESP IDF. I do not have access to a recent MACOS version, so I cannot replicate your situation.
Looks like another case of #17560. It seems there's an incompatibility between macOS's USB stack and Espressif's patched TinyUSB implementation.
@gohai Can you please modify
ports/esp32/main/idf_component.ymlwith the following change and see if things change? There have been new releases of both the patched TinyUSB and its relevant support code in the meantime since #17560 was filed.diff --git i/ports/esp32/main/idf_component.yml w/ports/esp32/main/idf_component.yml index f7773f4f4..728a0f84a 100644 --- i/ports/esp32/main/idf_component.yml +++ w/ports/esp32/main/idf_component.yml @@ -4,7 +4,8 @@ dependencies: espressif/esp_tinyusb: rules: - if: "target in [esp32s2, esp32s3]" - version: "~1.0.0" + version: "^1.7.6~1" + espressif/tinyusb: "^0.18.0~4" espressif/lan867x: version: "~1.0.0" rules:
@robert-hh the offending commit is d737112
Edit:
#17560 features mpremote losing data and crashing the board - here it duplicates data instead. @gohai did the board crash after the first file upload?
Just to be sure, does the
tusb_serial_deviceexample from ESP-IDF (inexamples/peripherals/usb/device/tusb_serial_device) still exhibit the same issues on macOS?Given the extreme difference in invalid behaviours for the same scenario (and now when entering text as well, which wasn't reported before), if Espressif's USB example works fine I'm starting to wonder if it's the current TinyUSB integration (either
ports/esp32/usb.[ch],shared/tinyusb, or both) that works only with the specific version that is optionally fetched as a submodule and needs changes for more recent versions.A friend of mine lent me their old macbook pro running macOS 13.7.6 in order to see whether I could reproduce the issue.
On an ESP32S3-DevKitC-R8N16, building the latest
masterusing ESP-IDF 5.4.2 (BOARD=ESP32_GENERIC_S3 VARIANT=SPIRAM_OCT) with those changes inidf_component.yml, and using that firmware made the board crash after 35 back-to-back transfers (mac → board, board → mac, diff; so actually 70 USB operations in a row) of a 250KBytes file (for i in $(seq 0 50); do mpremote cp file :file-$i; mpremote cp :file-$i file-$i; diff file file-$i; done).I have to free some space on my desk to connect both my regular dev machine and this laptop to each USB-C port of the board to run this under JTAG and see exactly what is happening on the board.
I don't see any issues with the console though, no characters lost or repeated. @robert-hh about the USB console echoes you reported, can you please give a bit more information on that? Like MicroPython version, ESP-IDF version, platform you're connecting from, board details, etc. Just to see if they're related to the commit in question or not.
v1.26 with the patch from #17865 (comment) appears to solve the issue for me on these two boards I tested with: LOLIN S2 Pico (USB-OTG with 4 MB Flash, 2 MB PSRAM), Waveshare S3 Zero (USB-OTG with 4 MB Flash, 2 MB PSRAM).
I also tried a stock v1.26 (without the patches) with an ESP32S3-S3-DEV-KIT-NXR8, similarly to the one you were testing with @agatti. There, I witnessed some repl hangs. I haven't tried running a modified firmware on this one yet.
Happy to do any additional testing.
about the USB console echoes you reported, can you please give a bit more information on that? Like MicroPython version, ESP-IDF version,
At the moment I can reproduce it only with the ESP32C6 and ESP32C3 with the actual MP versions. When connecting after power up I get strange errors in REPL, which complain about strings supplied to the input, which are in fact parts of the bootup message. There was a longer discussion with @projectgus about this phenomenon at other ports as well, like here #15298, which mentions as well #15158 (comment) in the NRF port. Both the RP2 and NRF port seem fine now.
I tested with plain v1.26.0 and copying larger files (10kB to 100 kB) aborts with
Syntax errormessage, copying from MAC to various ESP32-S3 modules.I tested again with latest MicroPython v1.27.0-preview.14.g4614ee9e6.dirty and the patch from above applied, i.e. this change:
diff --git a/ports/esp32/main/idf_component.yml b/ports/esp32/main/idf_component.yml index f7773f4f4..728a0f84a 100644 --- a/ports/esp32/main/idf_component.yml +++ b/ports/esp32/main/idf_component.yml @@ -4,7 +4,8 @@ dependencies: espressif/esp_tinyusb: rules: - if: "target in [esp32s2, esp32s3]" - version: "~1.0.0" + version: "^1.7.6~1" + espressif/tinyusb: "^0.18.0~4" espressif/lan867x: version: "~1.0.0" rules:
Compiling shows that the following versions are used:
NOTICE: Processing 4 dependencies: NOTICE: [1/4] espressif/esp_tinyusb (1.7.6~1) NOTICE: [2/4] espressif/mdns (1.1.0) NOTICE: [3/4] espressif/tinyusb (0.18.0~4) NOTICE: [4/4] idf (5.4.2)The problem disappears, i.e. I can copy large files now with no problem, tested with 10 kb to 1.5MB files.
This change to tinyusb likely fixed the problem: hathach/tinyusb#3127
Thank you!
Reacted by Angus GrattonThe patch from #17865 (comment) on v1.26.0 did not change the hangs in repl I was seeing on an ESP32S3-S3-DEV-KIT-NXR8.
- v1.25: works, no file corruption
- v1.26: hangs in repl, e.g.
help()will stop afterWelcome to MicroPython on the ESP32!and not accept any more input - v1.26 + patch: hangs in repl (same behavior)
This might of course also be an unrelated issue. The patch does seem to do the trick for for the two USB-OTG S2 and S3 modules I tested with.
about the USB console echoes you reported, can you please give a bit more information on that? Like MicroPython version, ESP-IDF version,
At the moment I can reproduce it only with the ESP32C6 and ESP32C3 [...]
Thanks, @robert-hh the issue you're seeing is not related to this (neither the C6 nor the C3 have a full-featured USB interface).
- added a commit that references this issue
on Aug 13, 2025 - linked a pull request that will close this issueesp32: Bump esp_tinyusb component, add component version lock files to source control. #17960
on Sep 9, 2025
Port, board and/or hardware
esp32-s3 (on a ESP32-S3 (QFN56) (revision v0.2), with 4 MB flash)
MicroPython version
MicroPython v1.26.0-preview.527.g593ae04ee on 2025-08-07; Generic ESP32S3 module with ESP32S3
mpremote 1.25.0, on macOS
Reproduction
mpremote cp bar.py :/bar.py)mpremote cp :/bar.py bar2.py)diff -u bar.py bar2.py)The files reproducibly differ for me. (Seeing syntax errors in repl as well, so the corruption doesn't seem to be caused by the second transfer back to the computer.)
Expected behaviour
No file corruption. (Content should be identical.)
Observed behaviour
File corruption, e.g.
def create_wav_header(sampleRate, bitsPerSample, num_channels, num_samples): datasize = num_samples * num_channels * bitsPerSample // 8 o = bytes("RIFF", "ascii") # (4byte) Marks file as RIFF o += (datasize + 36).to_bytes( 4, "little" + ) # (4byte) File size in bytes excluding this and Rittle" ) # (4byte) File size in bytes excluding this and RIFF marker o += bytes("WAVE", "ascii") # (4byte) File type o += bytes("fmt ", "ascii") # (4byte) Format Chunk Marker o += (16).to_bytes(4, "little") # (4byte) Length of above format data + o += (1).to_bytes(2, "little") # (2byte) Format tyat data o += (1).to_bytes(2, "little") # (2byte) Format type (1 - PCM) o += (num_channels).to_bytes(2, "little") # (2byte) o += (sampleRate).to_bytes(4, "little") # (4byte) o += (sampleRate * num_channels * bitsPerSample // 8).to_bytes(4, "little") # (4byte) + o += (num_channels * bitsPerSample /, "little") # (4byte) o += (num_channels * bitsPerSample // 8).to_bytes(2, "little") # (2byte) o += (bitsPerSample).to_bytes(2, "little") # (2byte) o += bytes("data", "ascii") # (4byte) Data Chunk Marker o += (datasize).to_bytes(4, "little") # (4byte) Data size in bytes return oAdditional Information
No, I've provided everything above.
Code of Conduct
Yes, I agree