Let user define cache location

@ejeschke

Original ask: Because we are not running astropy in someone's home directory, I want to be able to set the download/cache directory to someplace different than ~/.astropy/downloads. How do I do that? Setting environment variables (e.g., $HOME or $XDG_CONFIG_HOME) is not an option because that may affect other apps in unintended ways.

Might be related:

cc @ejeschke

@ejeschke

Naively, I would suggest we add an option via the config file like

[caching]
directory = my/cache/path

(naming is tricky here because section names normally correspond to subpackages, but here it would be "config", which is both uninformative and potentially misleading)
Note that with get_cache_dir and get_config_dir's implementations now being coupled (#17188), it'd be preferable to go with an implementation that makes sense for the configuration directory too, which can obviously not be configured via the configuration file itself, which leads me to the second option:
we could also support our own environment variables: ASTROPY_CONFIG_DIR and ASTROPY_CACHE_DIR, which would significantly reduce the chance of collision with another app. How about that, @ejeschke ?

FWIW I think #8748 and #9557 are orthogonal to this request.

docs already recommend XDG_CACHE_HOME: (https://docs.astropy.org/en/latest/config/index.html#getting-started)

If for whatever reason your $HOME/.astropy directory is not accessible (i.e., you have astropy running somehow as root but you are not the root user), the best solution is to set the XDG_CONFIG_HOME and XDG_CACHE_HOME environment variables pointing to directories, and create an astropy directory inside each of those. Both the configuration and data download systems will then use those directories and never try to access the $HOME/.astropy directory.

and various discussions in the past:
#6803 #10244 #10738

but seems XDG_CACHE_HOME is actually not used...
https://github.com/astropy/astropy/pull/17188/files#r1873758135

but seems XDG_CACHE_HOME is actually not used...
https://github.com/astropy/astropy/pull/17188/files#r1873758135

so bug was introduced by #16997, #17514 proposes a fix.

@neutrinoceros

The problem with the XDG_CACHE_HOME solution is that it affects many other applications that use this generic mechanism. I would propose a solution such as advocated above by @neutrinoceros where one could precisely specify the location using the astropy config machinery. In particular, to be able to do something like:

from astropy import config
...
cfg = config.get_config('caching')
cfg.directory = "/my/cache/path"

Apologies that I didn't research the config system thoroughly to know if this is the correct type of call. Of course, the value could default to XDG_CACHE_HOME/.astropy . But the point is that you would be able to precisely override it for cases where you want to.

Sure, having a specific way to configure locations might be useful in some cases, #10738 was proposing ASTROPY_CONFIG_DIR and ASTROPY_CACHE_DIR but it has not been implemented (yet, I guess just waiting for someone to do it).
However it has to be through environment variables since astropy uses config items when it is imported, and not sure if the cache location can be changed afterwards.

@ejeschke

I'm game to implement these env variables, but I'll let @ejeschke confirm that this path is satisfactory first.

Implementing the astropy environment variables might not be a bad idea, but it will be some time before the next feature release. In the meanwhile, it might be useful to set the XDG environment variables only for the process involving astropy, e.g.

$ XDG_CACHE_HOME=~/some_custom_dir ipython

It is also possible to use set_temp_cache and set_temp_config. They are not meant for global or permanent configuration, but using them as context managers might be good enough for many applications.

it will be some time before the next feature release

6 months will arrive sooner than we realize... 😸

Or 😰 .... we switch, like scipy and other major libraries, to managing downloads with pooch, which has very customizable cache directories and simultaneously eliminate a lot of our utils related to downloading (which IMO are rightfully in utils because they are not strictly in this project's scope and thus prime targets for replacement by well-documented, supported, and more feature-rich options).

Re: pooch -- That is a separate can of worms, see astropy/astropy-data#87 for a start.

@nstarman

FWIW I have a little bit of experience with pooch (which is used in a single function in yt, yt.load_sample), and I never had any reason to complain about it over 4 years that it's been in use there. But it's also not a crucial or even required dependency in that context, and if I understand this discussion correctly, pooch would be a hard requirement in astropy. Is this right, @nstarman ?

Yes. for replacing the downloading utils (probably actually make them a thin wrapper).
I've also been using pooch for a few years now and it's excellent.

Please move pooch discussion to a new issue (#17566) as that isn't the original ask here. Thanks.

TST: add tests for astropygh-17507

@ejeschke

@ejeschke , do you want to give the solution merged via #19575 a try? It should have landed on astropy nightly wheel by now. Would be nice to have your feedback before we cut v8.0.0 release so we can be sure it meets your needs. Thanks!

Navigation Menu

Repository navigation

Let user define cache location #17507

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Let user define cache location #17507

Description

Activity

neutrinoceros commented on Dec 6, 2024

saimn commented on Dec 6, 2024

saimn commented on Dec 6, 2024

saimn commented on Dec 6, 2024

saimn commented on Dec 6, 2024

ejeschke commented on Dec 8, 2024

saimn commented on Dec 9, 2024

neutrinoceros commented on Dec 9, 2024

eerovaher commented on Dec 9, 2024

pllim commented on Dec 11, 2024

nstarman commented on Dec 11, 2024

pllim commented on Dec 11, 2024

neutrinoceros commented on Dec 12, 2024

nstarman commented on Dec 12, 2024

pllim commented on Dec 12, 2024

pllim commented on Apr 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions