[Bug]: rspack build gets stuck at ci #9665
Description
System Info
Binaries:
Node: 22.13.1 - ~/.volta/tools/image/node/22.13.1/bin/node
Yarn: 1.22.19 - ~/.volta/tools/image/yarn/1.22.19/bin/yarn
npm: 10.8.2 - ~/.volta/tools/image/npm/10.8.2/bin/npm
Watchman: 2024.08.12.00 - /opt/homebrew/bin/watchman
npmPackages:
@rspack/cli: ^1.2.8 => 1.2.8
@rspack/core: ^1.2.8 => 1.2.8
@rspack/plugin-react-refresh: ^1.0.1 => 1.0.1
Details
to create a prod build I use a command WEBPACK_BUNDLE=production NODE_OPTIONS=--max-old-space-size=8196 rspack build, which basically runs rspack build. After upgrade from v1.1.6 to 1.2.8 (and it actually happened with 1.2.5) this command gets stuck and does not provide any additional info in 20min, then CI fails by timeout. Usual time to build static resources at CI is 100s.
I don't have a link with reproduction, as it happens from time to time. But here is a link to the GHA job
Reproduce link
No response
Reproduce Steps
WEBPACK_BUNDLE=production NODE_OPTIONS=--max-old-space-size=8196 rspack build
Activity
JSerFeng commentedon Mar 14, 2025
The reasons for stuck are so many, if you can provide a repro I believe we can help solving that in a few days
github-actions commentedon Mar 14, 2025
Hello @uladzimirdev, sorry we can't investigate the problem further without reproduction demo, please provide a repro demo by forking rspack-repro, or provide a minimal GitHub repository by yourself. Issues labeled by
need reproductionwill be closed if no activities in 14 days.uladzimirdev commentedon Mar 14, 2025
@JSerFeng clear, I'll try to do it, but so far no luck. Any tips on how to collect debug info?
hardfist commentedon Mar 14, 2025
@uladzimirdev there're some known deadlock potential bugs we're fixing, we'll release a canary version so you can try whether it fixes your problem
uladzimirdev commentedon Mar 14, 2025
thanks @hardfist. I'm trying to find the prerequisites to repro this possible deadlock. You know, we run 2 jobs at CI in parallel (only env variables are different) and second job gets stuck maybe 3/100 times, so it's really hard to verify atm.
it's not reproducible locally, no matter how much resources do I provide and how many other tasks I run to make CPU busy.
I've created CPU profiles, but nothing caught my eye. if it was a circular dep problem, I'd be able to reproduce it locally.
I had to upgrade swc packages together with rspack, maybe there can be some problem.
Rsdoctor didn't show any specific problem
CPU Profile
hardfist commentedon Mar 14, 2025
normally deadlock is caused by rust side so you need rust side profile to debug, you can generate cpu profile by following this guide https://rspack.dev/contribute/development/profiling#samply
you can try @rspack-canary/[email protected] by following this guide to see whether it solves your deadlock problem
uladzimirdev commentedon Mar 28, 2025
I've been able to gather some logs from github actions, using TRACE level. The file is huge, I uploaded it to google drive.
Please let me know if I need to upload it somewhere else so you have access to it
hardfist commentedon Mar 28, 2025
en, seem stuck in emit_assets phase(which is the last phase of rspack)
hardfist commentedon Mar 28, 2025
@uladzimirdev this may caused by https://github.com/web-infra-dev/rspack/pull/9587/files#diff-3fbb9f7dfbadceab7b0c89038c54de9851f25ae957a1f91aea05b0e46da2b209L367 which cause block_on on js function call and should be fixed in 1.3.0, can you help test whether it's still stuck with 1.3.0
slorber commentedon Mar 28, 2025
On Docusaurus repo (currently Rspack 1.2.5) we also encounter this.
Example CI job timeout: https://github.com/facebook/docusaurus/actions/runs/13945874964/job/39032844416
I'm not sure when it started to happen, but probably after 1.2.x, when we also turned on persistent cache.
I've also encountered it locally. I'm not 100% sure but I think it also happened with the dev server.
From what I remember, restarting the dev server or the prod build would consistently lead to the bundling process being stuck again at the end (100% progress bar, I think the status was "sealing" or "emiting" or something)
I also think that cleaning
node_module/.cachefolder permitted us to restart the dev server / prod build and make it complete successfully. So I assume it might be related to some kind of corrupted persistent cache?I'm not 100% sure of all this. Will try to investigate more the next time I see this issue locally, but I don't know exactly how to trigger it, unfortunately.
uladzimirdev commentedon Apr 1, 2025
@hardfist @h-a-n-a FYI I didn't have any deadlock after upgrade to 1.3.0, maybe it's too early, but my issue seems to be resolved
17 remaining items
hardfist commentedon Apr 28, 2025
can you help upload trace.json to github artifacts and share it with me?
jtsorlinis commentedon May 9, 2025
@hardfist sorry for not getting back to you, as this was holding up our development I ended up just moving us back to webpack+swc-loader for the time being.
I will try provide you with a trace when I get some spare time
benjdlambert commentedon May 10, 2025
@hardfist I would try and help here with a
trace.jsonas our builds fails all of the time, and feels pretty good test environment. However, from what I can tell at the moment, that the runner just totally crashes when running, so I'm not 100% sure how we're gonna be able to get a dump from a host that's unresponsive.I'm using
rspackthrough the docusaurus build, so is there a way that I can get atrace.jsonto disk easily? @slorber also maybe you might know of any props that get passed through torspackto be able to debug this?hardfist commentedon May 12, 2025
@benjdlambert does deadlock happens in backstage? if it's open source I can debug in your repo, honestly it's not easy to debug deadlock problems right now, we're still investigating better solutions for deadlock detection #10327
benjdlambert commentedon May 12, 2025
I'm not sure it's a deadlock, but it consistently fails every build and seems like the agent dies. backstage/backstage#29413 is the PR and branch. If I can help, please let me know 🙏
slorber commentedon May 12, 2025
@benjdlambert I'd suggest resolving to the very latest version of Rspack instead of
^1.3.0(your site is resolving to 1.3.4). I don't have any issues anymore. But maybe that's a different problem.Your logs do not clearly show that it's a problem related to Rspack, so I think it's too early to involve the Rspack team. Let's discuss that on your own repo instead. Here's a PR where I can help you figure it out: backstage/backstage#29905
hardfist commentedon May 12, 2025
if anyone have reproducible deadlock repo, please let me know, it would be very helpful for me to debug the deadlock cause
hardfist commentedon May 13, 2025
@benjdlambert @jtsorlinis @uladzimirdev can you share your rspack config file with me, I notice there maybe some deadlock risk if you use function in your rspack config( related to #10341)
uladzimirdev commentedon May 13, 2025
@hardfist https://github.com/metabase/metabase/blob/master/rspack.config.js
github-actions commentedon May 28, 2025
Since the issue was labeled with
need reproduction, but no response in 14 days. This issue will be closed. Feel free to comment and reopen it if you have any further questions.github-actions commentedon Jun 12, 2025
Since the issue was labeled with
need reproduction, but no response in 14 days. This issue will be closed. Feel free to comment and reopen it if you have any further questions.jtsorlinis commentedon Jul 19, 2025
Looks like one of the versions between 1.4.0 and 1.4.8 fixed the issue. Just tried updating to 1.4.8 and the builds are now working!