Skip to content

Client support for HTTP 1.1 persistent connections throughout the standard library #53949

Open
Open
Client support for HTTP 1.1 persistent connections throughout the standard library#53949
Assignees
orsenthil
Labels
stdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@ipatrol
ipatrol
mannequin
opened on Sep 2, 2010
BPO 9740
Nosy @loewis, @amauryfa, @orsenthil, @pitrou, @tpn, @ipatrol, @vadmium

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = None
created_at = <Date 2010-09-02.11:56:50.395>
labels = ['type-feature', 'library']
title = 'Client support for HTTP 1.1 persistent connections throughout the standard library'
updated_at = <Date 2015-04-11.02:53:59.744>
user = 'https://github.com/ipatrol'

bugs.python.org fields:

activity = <Date 2015-04-11.02:53:59.744>
actor = 'martin.panter'
assignee = 'orsenthil'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2010-09-02.11:56:50.395>
creator = 'ipatrol'
dependencies = []
files = []
hgrepos = []
issue_num = 9740
keywords = []
message_count = 11.0
messages = ['115365', '115367', '115368', '115369', '116641', '201875', '221299', '221304', '234609', '235999', '240459']
nosy_count = 8.0
nosy_names = ['loewis', 'amaury.forgeotdarc', 'orsenthil', 'pitrou', 'trent', 'ipatrol', 'martin.panter', 'vklogin']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue9740'
versions = ['Python 3.3']

Activity

  1. ipatrol commented on Sep 2, 2010

    @ipatrol
    ipatrolmannequin
    on Sep 2, 2010
    MannequinAuthor

    HTTP 1.1 introduced persistent connections nearly six years ago. Yet this resource saving and speed improving option is not available in the standard library. Can this be added?

  2. ipatrol
    added
    stdlibStandard Library Python modules in the Lib/ directory
    type-featureA feature request or enhancement
    on Sep 2, 2010

  3. amauryfa commented on Sep 2, 2010

    @amauryfa
    amauryfa
    on Sep 2, 2010
    Contributor

    It seems that httplib is exactly what you need:
    http://docs.python.org/library/httplib.html#examples

  4. ipatrol commented on Sep 2, 2010

    @ipatrol
    ipatrolmannequin
    on Sep 2, 2010
    MannequinAuthor

    No, httplib actually creates a second connection with the same object. Neither is their support in urllib, urllib2, nor in any of the HTTP servers. This would be really useful for a bot connected to an API.

  5. pitrou commented on Sep 2, 2010

    @pitrou
    pitrou
    on Sep 2, 2010
    Member

    Do you think you could provide a patch?

  6. ipatrol commented on Sep 16, 2010

    @ipatrol
    ipatrolmannequin
    on Sep 16, 2010
    MannequinAuthor

    Possibly, but I don't really have expertise in the underbelly of the HTTP system.

  7. orsenthil
    self-assigned this
    on Mar 19, 2011

  8. vadmium commented on Nov 1, 2013

    @vadmium
    vadmium
    on Nov 1, 2013
    Member

    I wrote a basic “urllib.request” handler class that I have been using for HTTP persistent connections. It is called PersistentConnectionHandler; see

    https://github.com/vadmium/python-iview/blob/80dc1b4/iview/hds.py#L442

    I am happy for this to be used as the basis for a patch for Python, or elsewhere.

    It manages a single connection. You pass an instance to urllib.request.build_opener(), and then it connects to whatever host is specified when open() is called. Any old connection is closed when you ask for a URL on a new host.

  9. vklogin commented on Jun 22, 2014

    @vklogin
    vkloginmannequin
    on Jun 22, 2014
    Mannequin

    Verified that Persistent Connections per HTTP 1.1 spec is already implemented and working correctly.

    See:
    http://hg.python.org/cpython/file/3f3de8c47ff8/Lib/http/client.py#l432

    Also, tested this in packet sniffer to verify that connection is being reused.

    But the server.py has in issue where the keep-alive can be used in HTTP/1.0:
    http://hg.python.org/cpython/file/7417d8854f93/Lib/http/server.py#l331
    So, the and in this line needs to be an 'or'.

  10. loewis commented on Jun 22, 2014

    @loewis
    loewismannequin
    on Jun 22, 2014
    Mannequin

    Mr. Kumar: the "and" is intentional. The server will use keep-alive messages only if it operates in HTTP/1.1 mode itself (protocol_version). By default, it operates in HTTP/1.0 mode, and does not enable persistent connections there. The initial implementation did, but it failed since it often would not send content-length headers, which are mandatory for persistent connections.

  11. vadmium commented on Jan 24, 2015

    @vadmium
    vadmium
    on Jan 24, 2015
    Member

    See bpo-3566 about tweaking the “http.client” module’s BadStatusLine handling to be more helpful when implementing persistent connections. I am dumping some thoughts here about persistent connections with the “http.client” module, gained by working on that bug.

    • Lib/xmlrpc/client.py appears to have persistent connection support, so may be useful for this bug.

    • RFC 7230 §6.5 <https://tools.ietf.org/html/rfc7230#section-6.5\> mentions monitoring for connection closure. This could be be partly implemented inside HTTPConnection by polling for closure before sending a request, but to fully implement might require the co-operation of the user calling into the module to check for closure at other times using select() or similar.

    • Current “http.client” assumes that each socket.makefile() object will not buffer any data from a subsequent response. Unsure if this is a problem in the real world, but I ran into it implementing test cases. E.g. if the server anticipates the first few bytes of the subsequent response:

    c.send(b"HTTP/1.1 200 Okay\r\nContent-Length: 0\r\n\r\n" b"HTTP/")

    then the client misses the “HTTP/” and raises BadStatusLine("1.1 200 Okay\r\n").

  12. vadmium commented on Feb 14, 2015

    @vadmium
    vadmium
    on Feb 14, 2015
    Member

    Opened bpo-23377 about losing the buffer at the end of a response

  13. vadmium commented on Apr 11, 2015

    @vadmium
    vadmium
    on Apr 11, 2015
    Member

    Tweaking the title to exclude servers. Persistent connections have apparently been supported in the low-level server for ages, since bpo-430706, and now that the close_connection flag is documented (bpo-23410), someone implementing a server should be able to use that to implement HTTP 1.0 keep-alive connections, etc as well.

    For the client aspect, this bug is rather vague about exactly what should be added. bpo-3566 has been committed and closed, meaning now we should have an easier way to detect when a persistent connection has been dropped by the server. Here is a list of other things that come to mind:

    1. Allow a way of polling for an unsolicited disconnection or 408 response without sending a request, either by:
    • exposing the sock attribute or adding a fileno() method, so that select() can be called, or
    • adding a new poll_response() or similar method
    1. Poll if disconnected before sending a request, to avoid in 99% of cases the problem with non-idempotent requests (like POST) where you do not know if they were accepted or not
    2. Change urlopen() to reuse the same connection when retrying a request (e.g. for redirects, authentication)
    3. A urllib.request.PersistentConnectionHandler class or other urlopen() level API where the caller can reuse the same connection for multiple requests to the same host. Like HTTPConnection mainly does, but with the extra redirect, error, etc handling of urlopen().

    I think the first point is important to do, because I have had to work around this by relying on the undocumented “HTTPConnection.sock” attribute. Once point 1 is done, point 2 might be easy to do in user code.

    The last two points I currently don’t have so much personal interest in seeing in the standard library, unless others also want something like them.

  14. vadmium
    changed the title [-]Support for HTTP 1.1 persistent connections throughout the standard library[/-] [+]Client support for HTTP 1.1 persistent connections throughout the standard library[/+] on Apr 11, 2015
  15. ezio-melotti
    transferred this issue fromon Apr 10, 2022
  16. serhiy-storchaka
    added this to HTTP issueson Jul 3, 2026
  17. github-project-automation
    moved this to Todo in HTTP issueson Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

stdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions