Repository navigation
Client support for HTTP 1.1 persistent connections throughout the standard library #53949
Description
| BPO | 9740 |
|---|---|
| Nosy | @loewis, @amauryfa, @orsenthil, @pitrou, @tpn, @ipatrol, @vadmium |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/orsenthil'
closed_at = None
created_at = <Date 2010-09-02.11:56:50.395>
labels = ['type-feature', 'library']
title = 'Client support for HTTP 1.1 persistent connections throughout the standard library'
updated_at = <Date 2015-04-11.02:53:59.744>
user = 'https://github.com/ipatrol'bugs.python.org fields:
activity = <Date 2015-04-11.02:53:59.744>
actor = 'martin.panter'
assignee = 'orsenthil'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2010-09-02.11:56:50.395>
creator = 'ipatrol'
dependencies = []
files = []
hgrepos = []
issue_num = 9740
keywords = []
message_count = 11.0
messages = ['115365', '115367', '115368', '115369', '116641', '201875', '221299', '221304', '234609', '235999', '240459']
nosy_count = 8.0
nosy_names = ['loewis', 'amaury.forgeotdarc', 'orsenthil', 'pitrou', 'trent', 'ipatrol', 'martin.panter', 'vklogin']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue9740'
versions = ['Python 3.3']Metadata
Metadata
Assignees
Projects
- Status
Activity
ipatrol commentedon Sep 2, 2010
HTTP 1.1 introduced persistent connections nearly six years ago. Yet this resource saving and speed improving option is not available in the standard library. Can this be added?
amauryfa commentedon Sep 2, 2010
It seems that httplib is exactly what you need:
http://docs.python.org/library/httplib.html#examples
ipatrol commentedon Sep 2, 2010
No, httplib actually creates a second connection with the same object. Neither is their support in urllib, urllib2, nor in any of the HTTP servers. This would be really useful for a bot connected to an API.
pitrou commentedon Sep 2, 2010
Do you think you could provide a patch?
ipatrol commentedon Sep 16, 2010
Possibly, but I don't really have expertise in the underbelly of the HTTP system.
vadmium commentedon Nov 1, 2013
I wrote a basic “urllib.request” handler class that I have been using for HTTP persistent connections. It is called PersistentConnectionHandler; see
https://github.com/vadmium/python-iview/blob/80dc1b4/iview/hds.py#L442
I am happy for this to be used as the basis for a patch for Python, or elsewhere.
It manages a single connection. You pass an instance to urllib.request.build_opener(), and then it connects to whatever host is specified when open() is called. Any old connection is closed when you ask for a URL on a new host.
vklogin commentedon Jun 22, 2014
Verified that Persistent Connections per HTTP 1.1 spec is already implemented and working correctly.
See:
http://hg.python.org/cpython/file/3f3de8c47ff8/Lib/http/client.py#l432
Also, tested this in packet sniffer to verify that connection is being reused.
But the server.py has in issue where the keep-alive can be used in HTTP/1.0:
http://hg.python.org/cpython/file/7417d8854f93/Lib/http/server.py#l331
So, the and in this line needs to be an 'or'.
loewis commentedon Jun 22, 2014
Mr. Kumar: the "and" is intentional. The server will use keep-alive messages only if it operates in HTTP/1.1 mode itself (protocol_version). By default, it operates in HTTP/1.0 mode, and does not enable persistent connections there. The initial implementation did, but it failed since it often would not send content-length headers, which are mandatory for persistent connections.
vadmium commentedon Jan 24, 2015
See bpo-3566 about tweaking the “http.client” module’s BadStatusLine handling to be more helpful when implementing persistent connections. I am dumping some thoughts here about persistent connections with the “http.client” module, gained by working on that bug.
Lib/xmlrpc/client.py appears to have persistent connection support, so may be useful for this bug.
RFC 7230 §6.5 <https://tools.ietf.org/html/rfc7230#section-6.5\> mentions monitoring for connection closure. This could be be partly implemented inside HTTPConnection by polling for closure before sending a request, but to fully implement might require the co-operation of the user calling into the module to check for closure at other times using select() or similar.
Current “http.client” assumes that each socket.makefile() object will not buffer any data from a subsequent response. Unsure if this is a problem in the real world, but I ran into it implementing test cases. E.g. if the server anticipates the first few bytes of the subsequent response:
then the client misses the “HTTP/” and raises BadStatusLine("1.1 200 Okay\r\n").
vadmium commentedon Feb 14, 2015
Opened bpo-23377 about losing the buffer at the end of a response
vadmium commentedon Apr 11, 2015
Tweaking the title to exclude servers. Persistent connections have apparently been supported in the low-level server for ages, since bpo-430706, and now that the close_connection flag is documented (bpo-23410), someone implementing a server should be able to use that to implement HTTP 1.0 keep-alive connections, etc as well.
For the client aspect, this bug is rather vague about exactly what should be added. bpo-3566 has been committed and closed, meaning now we should have an easier way to detect when a persistent connection has been dropped by the server. Here is a list of other things that come to mind:
I think the first point is important to do, because I have had to work around this by relying on the undocumented “HTTPConnection.sock” attribute. Once point 1 is done, point 2 might be easy to do in user code.
The last two points I currently don’t have so much personal interest in seeing in the standard library, unless others also want something like them.
[-]Support for HTTP 1.1 persistent connections throughout the standard library[/-][+]Client support for HTTP 1.1 persistent connections throughout the standard library[/+]