ENH: overhaul cache and config dir discovery by neutrinoceros · Pull Request #19575 · astropy/astropy

neutrinoceros · 2026-04-14T11:44:48Z

Description

This project is a deliverable on my current contract.

I've been working on this for a couple days. It isn't ready yet but I think I'm at the point where I want to see how it behaves in CI as I refine it.

~~based off #19559~~

To discuss:

~~should we also change the default cache/config location to match XDG standards ? (Why doesn't astropy use ~/.config/astropy by default (on Unix-like systems)? #6803)~~ (yes: this took a bit more effort because I didn't consider this idea soon enough, but the end result is more satisfying in more than one way)
~~should xdg variables only be supported on UNIX ?~~ (no: this was explicitly documented as cross-platform support before, no reason to change it)
~~do we actually want to look up environment variables every time get_cache_dir is called, or would it be enough to look once at import time ?~~ (probably not: it'd make testing much harder, which would be a shame for functions that are most useful in testing other code...)

(note that I ended up answering all of the above myself, though I'm still happy to take in any additional feedback)

TODO:

Notable differences between my new context managers and the existing APIs:

better names ("set" seems to imply "unset" isn't included, but it is...)
easily predictable behavior (ease of reasoning)
true isolation (symlinks are never created)
100% cross platform (windows isn't special cased)
safer defaults:
- temporary files and directories are not leaked by default (delete=True)
- pre-existing contents are never deleted (because the final dir is always created)
delete is keyword-only
dir_ (path) is positional only
support suffix and prefix
yield value is a Path (not a str)

About half the diff, at the time of opening, is shared between #19559 and a verbatim backport from the standard library that will be removed as soon as we drop support for Python 3.11¹

AI Discolure:
AI (Claude) was used in review (@astrofrog) to discover (and propose fixes to) 4 actual bugs in the implementation.
The design is entirely human and no code generation was used beyond drafting 3 regression tests associated with the aforementioned review.

By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

3.12.0 also gets patches from 3.12.1, so we'd have to require >=3.12.1 to actually remove the shim ↩

github-actions · 2026-04-14T11:45:01Z

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

Do the proposed changes actually accomplish desired goals?
Do the proposed changes follow the Astropy coding guidelines?
Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

neutrinoceros · 2026-04-16T09:27:20Z

I think I found a contradiction within the goals I originally set out to reach in this PR. I'll list them explicitly for clarity:

introduce safe alternatives to legacy set_temp_* context managers
make legacy and alternative APIs interoperable
avoid behavior changes in legacy APIs

The first point is reached, but I'm not sure it's possible (or worth the effort) to get 2) and 3)
Renouncing 2) would create a major flaw and discourage adoption (as a user of the alternative API, you have no way to guarantee that the legacy API will never be used by your dependencies). In practice, interoperability means that getters (get_(cache|config)_dir(|_path)) must retain correctness regardless what API was used to setup the current context.

It seems to me that an implication is that I must renounce 2), and in doing so I might also loose some of the benefit of having two competing APIs, though I expect not all of it (it's still nice to have an API that returns Path objects available, and changing the return type of existing functions is too much of a change to warrant doing it). I'll think about this problem some more.

neutrinoceros · 2026-04-17T11:13:22Z

I'm going to run with dropping 3). I'll push a couple separate commits with the following content

rewiring legacy code so that it uses the new design internally
adapt dedicated tests

I know that some other test will break as a result, but locally I only get about a dozen failures, and nothing that looks unredeemable, but I want to see if CI reveals any more.

astrofrog

This looks good to me now, modulo a small nit below.

From what I can tell I think the issue of the config being created in the home directory when running pytest might not be fully fixed, but I'll let you decide whether to address this here or in a follow-up PR. The cleanest fix might be to go back to defining the temporary directories in conftest.py.

I am approving this since I will not be working until Thursday and I think it's in a good enough shape to merge. Thank you for all your work on this @neutrinoceros !

I'll let you decide if you want to squash the commits, feel free to merge once you feel it's ready.

- conform to `XDG` specifications by default - enforce true isolation by never creating symlinks - avoid special-casing Windows - add support for `ASTROPY_(CACHE|CONFIG)_DIR` environment variables - add safer context managers for defining temporary cache and config directories - soft-deprecate `set_temp_cache` and `set_temp_config`

neutrinoceros · 2026-04-28T13:49:41Z

thanks a bunch Tom. I just squashed the history and will set auto-merge now, just so that it stops growing. I'll make sure to address any lingering issue with test pollution as a follow up PR, thought that one will not be that urgent since it could go in after the FF.

pllim · 2026-04-28T14:36:33Z

-    (see `here <https://wiki.archlinux.org/index.php/XDG_Base_Directory_support>`_
-    for more details), but ``astropy`` will use them on any OS as a more
-    general means to know where user-specific configurations should be written.
+    ``ASTROPY_CACHE_DIR`` takes precedence over ``XDG_CACHE_HOME``, and defines an


The formatting here is a bit weird when rendered:

https://docs.astropy.org/en/latest/environment_variables.html

Thanks for spotting this. I can never get rst right... I'll make sure to address it tomorrow !

Or you can review #19607 :)

pllim · 2026-04-28T15:05:12Z

My main concern with this PR, especially merged so close to feature freeze is that right off the bat, it silently changes where my cache and config goes. For years now, I have them here:

/home/user/.astropy/cache/
/home/user/.astropy/config/

Now when I install astropy with this patch:

astropy.config.get_cache_dir() --> /home/user/.cache/astropy/
astropy.config.get_config_dir() --> /home/user/.config/astropy/

There are no warnings. A typical user won't notice and now will have dead cache and config in the old location. And might wonder why their existing cache and config are no longer in effect.

The new session fixtures pollute the user's real $HOME

This is definitely a problem. My HOME quota is set by the IT department and is pretty small. So locally running astropy test suite filling up my HOME is very bad. This should be fixed before v8.0 is released. The best outcome is have pytest tmp_path control where the stuff is going, because just redirecting to /tmp is also not good enough in a shared system where user might not have write access to that directory.

pllim · 2026-04-28T15:13:55Z

For completeness, I opened a dedicated thread at https://groups.google.com/g/astropy-dev/c/G0cVLPbEl6M

neutrinoceros · 2026-04-28T15:25:19Z

There are no warnings. A typical user won't notice and now will have dead cache and config in the old location. And might wonder why their existing cache and config are no longer in effect.

this is perfectly addressable in a follow up PR and could even be merged beyond the feature freeze. We have a couple decisions to make when the old default path exists but the new one doesn't:

warn ?
fallback to the legacy directory ?

I think what's best depends whether we want to support legacy locations indefinitely or not. At the moment I don't think that'd be a huge maintenance burden so I can give it a try. If that's the path we take, I don't see a particularly convincing reason to emit a warning. Thoughts ?

The new session fixtures pollute the user's real $HOME
This is definitely a problem.

I addressed it already, and I'll double check tomorrow, but I want to clarify that the pollution in question was just about the default dir location being created if it didn't exist yet. The dir was not populated by a test session and just remained empty. It is a problem best addressed soon, but it's not nearly as bad as it seems, and it wouldn't blow away your HOME quotas.

pllim · 2026-04-28T15:36:45Z

We have a couple decisions to make

Hopefully there will be more opinions (than just mine) to help us decide. Philosophy of least surprise would be to use legacy dir but maybe not everyone wants that... 🤪

neutrinoceros · 2026-04-28T15:39:08Z

That's also what pip does btw:

the default is $HOME/.config/pip/pip.conf
the legacy default $HOME/.pip/pip.conf is still looked up if the current one is missing.

pllim · 2026-04-28T15:43:49Z

-:func:`~astropy.config.get_config_dir`::
+:func:`~astropy.config.get_config_dir_path`::

-    >>> from astropy.config import get_config_dir


Do we also want to mention get_config_dir? Sometimes you just want the string, not Path, and we already have a function for that.

if you want a string, str is builtin, so you don't even need an import. Even if we wanted to deprecate the str-returning function, it'd remain trivially supported

(long term I would like to deprecate them, which is why I'm not keen on recommending them in docs)

Is there a follow-up issue to deprecate them in the future?

No, but it's not a big deal if it doesn't happen, just an idea I might come back to in a while.

I opened #19609 so we don't forget about it. We can close it as won't fix if you change your mind. Thanks.

keflavich · 2026-04-28T15:45:28Z

👍 from me on the idea of warning users if their cache directory has changed. I also have a tightly limited $HOME drive, and astroquery caching (which uses the same system and is probably affected?) can result in large data dumps. A warning could be emitted at import time if the old cache directory exists but is not being used.

pllim · 2026-04-28T15:49:33Z


    # See what happens when set_temp_cache is pointed at a file
-    with pytest.raises(OSError):
+    with pytest.raises(Exception) as _:


Why change OSError to general Exception here and why do we need as _?

because the exact exception type is not in our control and changed with my rewrite. The as _ is needed to satisfy ruff, otherwise it'll flag the call to pytest.raises as imprecise because I don't use a match arg

neutrinoceros · 2026-04-28T16:17:20Z

A warning could be emitted at import time if the old cache directory exists but is not being used.

we can't warn at import time, because the config/cache dir can really be looked up at any point, not specifically at import, and change dynamically.

pllim · 2026-04-29T03:18:23Z

Just so we don't lose track, I opened #19611 that would block v8.0.0 release.

pllim · 2026-05-01T17:54:25Z

@keflavich we ended up falling back to "legacy" dirname if it exists, instead of issuing warning. Are you okay with this? Please let us know if it doesn't work as expected for you. Thanks!

keflavich · 2026-05-01T17:58:31Z

yes, that is great! Thanks for keeping things unbroken!

neutrinoceros added this to the v8.0.0 milestone Apr 14, 2026

neutrinoceros added the config label Apr 14, 2026

github-actions Bot added the testing label Apr 14, 2026

neutrinoceros force-pushed the enh/astropy-spec-envvars branch 13 times, most recently from 1c577a3 to f661f36 Compare April 15, 2026 15:20

neutrinoceros added Bug Effort-high Refactoring API change PRs and issues that change an existing API, possibly requiring a deprecation period labels Apr 15, 2026

neutrinoceros force-pushed the enh/astropy-spec-envvars branch 2 times, most recently from 666ff4b to ebde048 Compare April 16, 2026 09:18

neutrinoceros force-pushed the enh/astropy-spec-envvars branch 5 times, most recently from a99b136 to 1d9786e Compare April 17, 2026 09:43

astrofrog approved these changes Apr 28, 2026

View reviewed changes

Comment thread docs/changes/config/19575.api.rst Outdated

neutrinoceros force-pushed the enh/astropy-spec-envvars branch from 16e788e to 554df66 Compare April 28, 2026 13:47

neutrinoceros enabled auto-merge April 28, 2026 13:49

neutrinoceros merged commit 58824f4 into astropy:main Apr 28, 2026
34 of 37 checks passed

pllim reviewed Apr 28, 2026

View reviewed changes

pllim mentioned this pull request Apr 28, 2026

DOC: Fix formatting in what's new and env var #19607

Merged

1 task

pllim reviewed Apr 28, 2026

View reviewed changes

This was referenced Apr 28, 2026

API: Deprecate get_config_dir and get_cache_dir #19609

Open

Let user define cache location #17507

Closed

Remaining cache and config overhaul issues for v8.0.0 #19611

Closed

This was referenced Apr 29, 2026

API: respecting Windows-specific path conventions (%APPDATA% instead of $XDG_CONFIG_HOME) #19617

Open

ENH: fallback to legacy default paths if they exist #19618

Merged

API: avoid creating cache directories on discovery #19631

Merged

This was referenced May 3, 2026

BUG: 20 test failures when unsetting the HOME env var #19648

Closed

BUG: fix an incorrect conditional (missing method call) #19672

Merged

TST: future proof test session settings for astropy 8.0+ sunpy/sunpy#8616

Merged

neutrinoceros mentioned this pull request May 23, 2026

Tracking issue: future evolutions in cache and config discovery strategy #19803

Open

Conversation

neutrinoceros commented Apr 14, 2026 • edited by pllim Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Footnotes

Uh oh!

github-actions Bot commented Apr 14, 2026

Uh oh!

neutrinoceros commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neutrinoceros commented Apr 17, 2026

Uh oh!

astrofrog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

neutrinoceros commented Apr 28, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pllim commented Apr 28, 2026

Uh oh!

pllim commented Apr 28, 2026

Uh oh!

neutrinoceros commented Apr 28, 2026

Uh oh!

pllim commented Apr 28, 2026

Uh oh!

neutrinoceros commented Apr 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

keflavich commented Apr 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

neutrinoceros Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

neutrinoceros commented Apr 28, 2026

Uh oh!

pllim commented Apr 29, 2026

Uh oh!

pllim commented May 1, 2026

Uh oh!

keflavich commented May 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

neutrinoceros commented Apr 14, 2026 •

edited by pllim

Loading

neutrinoceros commented Apr 16, 2026 •

edited

Loading

neutrinoceros Apr 28, 2026 •

edited

Loading