Repository navigation
Custom cache location does not work when the directory does not exist #18436
Description
Description
Hello, I am trying to change the cache path as it keep exceeded disk quota when downloading data. I have set the environmental variable XDG_CACHE_HOME, but when checked with astropy.config.paths.get_cache_dir() it remains the default. How do I fix this?
Expected behavior
from https://docs.astropy.org/en/stable/environment_variables.html
XDG_CACHE_HOME
These environment variables control where data files are cached. Astropy will cache files in $XDG_CACHE_HOME/astropy. If not set, or if $XDG_CACHE_HOME/astropy does not exist, astropy will default to $HOME/.astropy/cache. See Downloadable Data Management (astropy.utils.data) for how to programmatically set or get the location of the corresponding directory at runtime.
How to Reproduce
Versions
import astropy
astropy.__version__# Paste the result here
'5.3.4'
Activity
pllim commentedon Jul 15, 2025
astropy 5.3.4 is very old. Are you able to upgrade to 7.1.0 and see if the problem is still there for you? I vaguely remember some bug fixes in this matter some time ago.
pllim commentedon Jul 15, 2025
p.s. You are not on Windows, are you?
danielhill676 commentedon Jul 15, 2025
Hi thanks for the reply, I've just updated now (didn't realise I was using an old version), however this doesn't seem like its fixed my issue as astropy.config.paths.get_cache_dir() continues to print the default home directory. This is on a linux machine, all run from inside the terminal.
pllim commentedon Jul 15, 2025
Then may you please elaborate how you set
XDG_CACHE_HOME? Is it sticky (via shell config)? Did you tryecho $XDG_CACHE_HOMEand make sure it stuck? I can try to replicate locally if I know your exact steps. If you post terminal output, please do remember to scrub out sensitive info before posting here. Thanks!danielhill676 commentedon Jul 15, 2025
I've tried to set it through various different means. In this test I set it with:
export XDG_CACHE_HOME=/data/path/to/desired/cache
This is verified with echo $XDG_CACHE_HOME which returns:
/data/path/to/desired/cache
This is then tested with opening a python prompt by typing 'python' into the command line and then:
The XDG_CACHE_HOME=/data/path/to/desired/cache line is also present in my bashrc file.
pllim commentedon Jul 15, 2025
Okay, here is the catch:
/data/path/to/desired/cache/astropydirectory must already exist.Here is another catch:
XDG_CACHE_HOMEis also used by the OS for other things, so you will see things likefontconfigdirectory created there, alongsideastropystuff. So depending on how things are set up on your end, you might have to ensure you copy other stuff over to the new location as well.So, for your immediate problem, please try this:
mkdir /data/path/to/desired/cache/astropyastropy.config.paths.get_cache_dir()again.pllim commentedon Jul 15, 2025
The check that confused both of us is here:
astropy/astropy/config/paths.py
Line 175 in 1f4bd60
I think @neutrinoceros and I talked about this a while back, but we decided not to change old behavior because it might silently break things that already work for others.
[-]Astroquery set cache[/-][+]Custom cache location does not work when the directory does not exist[/+]danielhill676 commentedon Jul 16, 2025
Unfortunately the /data/path/to/desired/cache/astropy directory has already been created and I've checked that the path definitely lines up with the one supplied to XDG_CACHE_HOME, so it seems something else is going on.
neutrinoceros commentedon Jul 16, 2025
I don't recall agreeing to this conclusion, though the status quo always has the upper hand (as it should), so that's entirely possible. However, I think this ticket is concrete evidence that the current behavior is confusing and error-prone, so I'd like that we seize the opportunity to re-assess our conclusion. To me, actual users running into problems feels more important than hypothetical backwards compatibility issues;
astropysub directory being needed isn't the problem, but the fact that we silently ignore user configuration if it doesn't is. I think anAstropyUserWarningis warranted in this case. Anything downstream that'd be broken by a warning should be informed that their configuration is ignored, and if they're turning warnings into errors, then it's pretty clear they want to know about this sort of stuff.pllim commentedon Jul 16, 2025
That reminds me of this PR that had a bunch of discussions but I do not have time to context switch back into it yet...
pllim commentedon Jul 16, 2025
Re: #18436 (comment)
Well, that is weird because on my end, as soon as I create that extra
astropysub-directory, it worked for me... Ah, wait, you saidastroquery? Did you also trymkdir astroquery?pllim commentedon Jul 16, 2025
Also is your
/data/path/to/desired/cache/astropya symlink?danielhill676 commentedon Jul 16, 2025
I think I'd misread your comment yesterday and had not created the
astropysubdirectory in /data/path/to/desired/cache , I've now done so and astropy.config.paths.get_cache_dir() now finds it. Thanks!