Repository navigation
Tmux hangs when it runs out of file descriptors #14
Description
Activity
I'm not sure the original suggestion of commenting out anything in server_accept_callback() is going to help. I see nothing in strace from errno being set---in fact. from tmux's point of view, everything is as it should be.
I was able to reproduce this problem on the machine you gave me access to---in fact, I've left tmux running there in that state for now. It looks to me like a potential bug in libevent---event_base_loop() seems to be spinning on poll() all the time, although I wouldn't know why, and don't have the time to check what libevent is doing.
Hey @ThomasAdam,
thanks for checking this.I forgot to mention, currently installed tmux on the test server has the
server_accept_callback()commented out and then compiled (dunno if that makes a difference for the investigation). Tmux source is git cloned to/root/src/tmux/dir.libevent does not handle POLLNVAL, so it is spinning. It should probably handle that, but we should probably not be adding the file descriptor in the first place (or not closing it without removing the event). I will look at that as soon as I figure out what fd it is.
This is not the problem on OS X, because we do not use poll on OS X (it is broken), we use select. It may be a similar issue because it looks like libevent does not check exceptfds for select, although on Linux everything works fine with select.
Try this please, it seems to fix Linux: https://goo.gl/V1zWZM
For jobs, we were calling bufferevent_disable(EV_READ) and closing our end of the file descriptor when the other end was closed by the child process, but that is clearly not enough to stop libevent adding it to the poll set. But we can't free the bufferevent because it still contains stdout data that we may want. So we need to leave it active and mark the job in a different way (with an explicit state).
Hi,
that patch fixes the issue on linux!I've applied the patch and recompiled on OS X and unfortunately the issue is still there :(
Not sure what is the best approach to handling this. Should we look into renting an OS X vps? They are less of a commodity than Linux vps's though.What does ktrace show on OS X when it is spinning?
Hi,
thanks for the work on this issue so far and sorry for waiting so long for the response.I wasn't able to find
ktraceprogram for OSX, but apparently there'sdtruss- adtracescript that is equivalent tostracetool on Linux. Here's the link todtrussprogram output for tmux process when it's spinning on 100% CPU on OSX.Also I managed to find a person that will give us access to OSX machine for free (as mentioned before, OSX servers aren't that cheap, they are in the range of ~US$ 50/month).
I got SSH access to this server via OpenVPN and managed to reproduce tmux issue there.Thomas, Nicholas are you interested in getting access to this OSX server?
I think "direct" debugging will speed things up. Also this person is willing to give long term access for the needs of tmux project.
If you think this is a good idea, I'll email you directly so we can setup OpenVPN + ssh access for you.Thanks
The logs indicate some condition in write_nocancel() leading to a EBADF on select() which is strange.
Sure, if I can have access, I can take a look. But it won't be for a while.
It is a similar problem, we need to find out what that file descriptor is and why it is being closed without removing the event.
If you give me ssh details I will take a look at some point.
Thanks for the willingness to look into this.
Unfortunately I can't give you direct ssh access to the OSX server - first setting up a VPN is needed.
Hopefully that's not too much effort. I'm not knowledgeable about VPNs but I was able to set it up quickly...I'll email you directly so a person providing the OSX server can send you VPN and ssh details.
I can probably dig out an OS X box at some point without having to faff with VPNs. Do you reproduce the same way as on Linux? Where do you get dtruss?
Ok, didn't know you had access to an OSX machine.
dtrusscomes on a mac by default, at least since version10.6(that was in 2009). Basic usage is pretty straightforward:sudo dtruss -p 'pid' > output.txt 2>&1(it tripped me that sudo seems to be required).Yes, the steps to reproduce are pretty much the same as on Linux:
-
tmux.confcontent:set -g prefix C-a set -g @success "yep" set -g status-right 'Script #(ping -c 1 www.google.com >/dev/null 2>&1 && printf "$(tmux show -gqv @success)")' -
start tmux
-
create many windows with
for ((i=1;i<=150;i++)); do tmux new-window; done -
now create a couple more windows manually with
prefix+cjust to make sure a limit is hit -
try
tmux show -g status-rightin one of the panes, there's no output -
at this point tmux process cpu goes to 100%
-
I think this fixes the main problem but I suspect there is something else wrong in there too, although I can't reproduce it consistently:
diff --git a/server-client.c b/server-client.c index c9c0c3e..7219ed7 100644 --- a/server-client.c +++ b/server-client.c @@ -95,6 +95,8 @@ server_client_create(int fd) environ_init(&c->environ); + c->cwd = -1; + c->cmdq = cmdq_new(c); c->cmdq->client_exit = 1;This fixes that and another problem http://pastebin.com/raw.php?i=ktrTjvEG
Hi,
thank you for working on a solution!I tried applying that patch onto latest git master, I got errors with this command
curl http://pastebin.com/raw.php?i=ktrTjvEG | git apply. In the end I just carefully made each change manually.Anyway, I can confirm this change fixes the issue on OSX. I can't reproduce the 100% CPU scenario anymore. Thank you for all the work and patience!
Hmm looks like pastebin messes up the line endings. Applied the patch now. Thanks.
FYI, https://gist.github.com/zchee/5d712d43d1993eb60629.
for Homebrew.patch :p1 do url "https://gist.githubusercontent.com/zchee/5d712d43d1993eb60629/raw/e8d1151b1b81234700fad30ef485160c826e30e5/tmux-hangs.diff" sha256 "d18e8a3b47d283fbbefb850369362127345117eacc5b8371fa8505061af7df08" endThis thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
- locked and limited conversation to collaborators
on Feb 17, 2020
Hi,
this is a reply to the original issue 196 opened on sourceforge.
The problem was spotted on OSX where the default file descriptors limit is 256. When tmux used all the file descriptors, it would hang and the CPU would go to 100%.
Below is the copy/paste of the excerpts from the previous discussion.
I was able to reproduce the issue on Linux too. Here are steps to reproduce:
test machine: Ubuntu 14.04 cloud VPS with 512Mb ram
tmux -V is 2.0 (also tested on latest version from git, behaves the same)
the default output of ulimit -n command is 1024. To "simulate" OS X default limit, this was reduced to 256 with the following command: echo 'ulimit -n 256' | sudo tee -a /etc/profile, and the machine was restarted
tmux.conf:
start tmux
the ping command in status-right works ok, displays "yep"
use this command to create large number of windows
for i in $(seq 1 150); do tmux new-window; donethere's no output from ping command now. At this point tmux starts using 100% CPU sometimes. By doing the next step that happens consistently.
In one of the panes run
tmux show -g status-right. The command returns no output and at that moment tmux process CPU usage goes to 100% and never goes down. However, tmux stays responsive, windows can be changed and executing other commands works ok (which was not the case on OSX where things just freeze).Even with the above steps, Nicholas wasn't able to reproduce the issue. The suggestion was to try the following:
So here's the update to this issue: unfortunately updating the above bit and recompiling didn't change anything. I can still get tmux to use 100% CPU consistently.
Idea for moving forward on this: what do you think about using the linux server where reproducing the issue is easy? Would that help?
I've created a linux server with everything set up (tmux installed, tmux.conf in place). Nicholas can ssh in with his keys and Thomas with his. IP is:
46.101.186.232. User isroot. Tmux is cloned in/root/src/tmux. Feel free to do whatever you want on the server, I'll nuke it once we're done debugging.Once again, steps to reproduce the issue consistently on the server:
for i in $(seq 1 150); do tmux new-window; donetmux show -g status-righthtopshould show tmux using 100% CPUI hope that will help!