Respect the max age cache control for simple API responses, with option to modify

In #5791 caching simple API responses was turned off, even if a max age cache control response was sent back, this solved one use case at the performance cost of everyone.

When doing multiple pip commands, especially ones that don't require downloading distributions, like pip lock, pip install --dry-run, and pip install where the distributions are already in cache, this contributes to a significant amount of the resolution time.

uv respects the the max age cache control and gives users a way to set their own values for the API and files: https://docs.astral.sh/uv/concepts/indexes/#customizing-cache-control-headers.

I would like to agree in principle we should do the same, and then we can agree on how to do it.

I'm happy to agree that in principle. We'll need a different UI to the one uv uses (I'm -1 on reading pip config from pyproject.toml) but we can decide on a UI later, once someone starts work on the feature.

Agreed, this should be a normal option(s), pip should only read from a pyproject.toml when the spec explicitly says it should, like dependency groups (or if #13003 is ever agreed upon and implemented).

I'm not working on this right now, so there may be issues I've not considered, but my general idea was something like --cache-control-api <str> --cache-control-files <str>, with examples of supporting the use case of uploading a new package and then installing it without having to wait for the cache to expire.

We don't currently have any semantics for per-index options, I didn't intend for this feature to solve that, I think that needs to be solved independently.

👍 on doing this

my general idea was something like --cache-control-api --cache-control-files , with examples of supporting the use case of uploading a new package

Is there a reason for not going with something like --force-metadata-refresh and using the :all: style for defining which projects[packages] should be refreshed? This seems to work quite well with uv.

Is there a reason for not going with something like --force-metadata-refresh and using the :all: style for defining which projects[packages] should be refreshed? This seems to work quite well with uv.

I'm open to ideas, but I don't understand what force refresh metadata means in this context? Could you describe what that means for what http requests?

The ability to manually set cache control, as I proposed, is taken from uv.

To force uv to revalidate cached data for all dependencies, pass --refresh to any command (e.g., uv sync --refresh or uv pip install --refresh ...).

This will then invalidate the http cache, even if still covered by the max-age (https://docs.astral.sh/uv/concepts/cache/). I wasn't aware of the cache-control-api, and the ability to set arbitrary max-ages (as per astral-sh/uv#15454 (comment)). This sounds a bit niche to me, but perhaps there are use-cases that really require that level of control. Honestly though, I would argue that setting a fake/bespoke max age setting is fruity and could be achieved just as well by somebody who knows what they are doing with a reverse proxy...

It is for this reason that I think the refresh type flag is good enough for the use case of getting a release that you just made to PyPI installed into a virtual environment.

Could you describe what that means for what http requests?

I think we are only talking about simple api metadata requests here, since package downloads are immutable and there should never be a need to refresh those (if you need to do that, just delete your cache directory 🤷‍♂️). So for me, this would entirely be about the /{project}/ simple endpoint (e.g. pypi.org/simple/{project}).