Repository navigation
Linux sockets perf: Reduce syscalls by being smart about handling partial reads/writes #43966
Description
Currently, for every send/receive operation, we always issue a syscall to see if we can perform the operation synchronously. If this fails (i.e. E_AGAIN), we enqueue the operation and wait for an epoll notification before trying again.
We can avoid this initial sync attempt for streaming protocols (i.e. TCP) in many cases by remembering whether the previous send/receive returned a partial response. That is, for receive, it didn't fill the entire buffer; for send, it didn't consume the entire buffer. In these cases we can know that the socket is now not ready and we need to wait for another epoll readiness notification. So on the next send/receive, we can detect that the socket is still not ready and skip the attempt to perform the operation synchronously.
Here is the relevant passage from Linux epoll man:
9. Do I need to continuously read/write a file descriptor until
EAGAIN when using the EPOLLET flag (edge-triggered behavior)?
Receiving an event from epoll_wait(2) should suggest to you that
such file descriptor is ready for the requested I/O operation.
You must consider it ready until the next (nonblocking)
read/write yields EAGAIN. When and how you will use the file
descriptor is entirely up to you.
For packet/token-oriented files (e.g., datagram socket, terminal
in canonical mode), the only way to detect the end of the
read/write I/O space is to continue to read/write until EAGAIN.
For stream-oriented files (e.g., pipe, FIFO, stream socket), the
condition that the read/write I/O space is exhausted can also be
detected by checking the amount of data read from / written to
the target file descriptor. For example, if you call read(2) by
asking to read a certain amount of data and read(2) returns a
lower number of bytes, you can be sure of having exhausted the
read I/O space for the file descriptor. The same is true when
writing using write(2). (Avoid this latter technique if you can‐
not guarantee that the monitored file descriptor always refers to
a stream-oriented file.)
@adamsitnik prototyped this in #38747 and it showed a nice win -- 5% on TechEmpower JSON. So this definitely seems worth doing.
In #38747 we ran into some issues with handling EOF properly -- i.e. EPOLLHUP and/or EPOLLRDHUP. I think these are solvable -- see discussion on that issue. We didn't have time to chase this down for 5.0.
@adamsitnik Any other thoughts/comments here?
Activity
tmds commentedon Oct 29, 2020
Though the man page says otherwise, I think receive may return with less bytes than the buffer size when there is a signal pending.
geoffkizer commentedon Nov 5, 2020
Is there any way we can confirm or deny that?
I've seen references in other projects about leveraging this optimization. (Of course, I can't find them at the moment...)
wfurt commentedon Nov 5, 2020
I thin receive can receive less than asked for when more data is awailable - but I would think that is rare case. If this is optimization, that can still work in our favor IMHO.
dotnet-policy-service commentedon Nov 12, 2025
Due to lack of recent activity, this issue has been marked as a candidate for backlog cleanup. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will undo this process.
This process is part of our issue cleanup automation.
dotnet-policy-service commentedon Nov 26, 2025
This issue will now be closed since it had been marked
no-recent-activitybut received no further activity in the past 14 days. It is still possible to reopen or comment on the issue, but please note that the issue will be locked if it remains inactive for another 30 days.