Repository navigation
BUG: config and cache dir configured via env variables are not (always) automatically created #18791
Description
This came up in #17934, specifically #17934 (comment)
The problem is that, if one defines e.g. XDG_CACHE_HOME to an existing dir that's also writable, but missing an astropy subdirectory, the env variable is currently silently ignored.
The user experience is already being improved in #17934, which adds a clear warning that the env var is ignored, and explains why.
However, we could actually try to create the missing subdirectory first, and only fallback to ignoring the env var if that doesn't work.
This issue is a reminder to implement this behavior as early as possible during the next cycle (presumably 8.0), as a follow up to #17934.
Activity
pllim commentedon Oct 28, 2025
Here is the behavior on
mainat the time of writing.Starting condition:
XDG_CACHE_HOMEis defined and the directory exists but$XDG_CACHE_HOME/astropydoes not exist.If ~/.astropy/cache does NOT exist
Calling
astropy.config.paths.get_cache_dir()would create~/.astropy/cacheand also create$XDG_CACHE_HOME/astropy. The latter is a symlink to the former. The function would only return the former.If ~/.astropy/cache already exists
Calling
astropy.config.paths.get_cache_dir()would simply return~/.astropy/cachewithout worrying aboutXDG_CACHE_HOMEat all, so$XDG_CACHE_HOME/astropyis not created.I think this issue is about the second "if" here.
So the question is: Would anyone actually want
$XDG_CACHE_HOME/astropyif~/.astropy/cachealready exist? This question is separate from #17507 .I also think this behavior, while confusing, is still consistent the following docs (as per v7.2)?
neutrinoceros commentedon Oct 28, 2025
Thanks, I always forget that the existence of
$HOME/.astropyalso plays a role here...I think that'd be the least surprising possible behavior for anyone setting
XDG_CACHE_HOME, yes. Env variables usually take precedence over any other global state. I'm saying "usually", but what I really mean is "in all cases I know of, other than this one weird behavior in astropy's config system".pllim commentedon Oct 28, 2025
I think for this particular issue, maybe it would be uncontroversial to create that symlink even if
~/.astropy/cachealready exists for v8.0neutrinoceros commentedon Oct 28, 2025
at the risk of starting a war, I actually don't think we should ever be creating symlinks 😬
pllim commentedon Oct 28, 2025
I am sure there was a reason (perhaps to prevent accidental multiple or conflicting copies?) but I wasn't part of the project when this module was designed.
[-]BUG: config and cache dir configured via env variables are not automatically created[/-][+]BUG: config and cache dir configured via env variables are not (always) automatically created[/+]andresgur commentedon Jan 27, 2026
if I my join this fun party all I can say is that I've wasted like 3h of my life trying to set up a new cache/config directory and not only I failed to do it, I also failed to understand how any of it works
It's also very confusing that
a) https://docs.astropy.org/en/stable/environment_variables.html#environment-variables
Says to create a folder $XDG_CONFIG_HOME/astropy for config, when it would make more sense to have astropy/config (as per the DEFAULT structure $HOME/.astropy/config). Initially I expected $XDG_CONFIG_HOME to be the config directory itself.
b) astropy.config.paths.get_cache_dir_path("astropy") this little bitch doesn't look for "astropy" it looks for ".astropy", which is not what the docs above say (meaning if you set something/astropy (without .) it won't work and good look figuring it out without any warnings)
At least that's what I grasped...
pllim commentedon Jan 27, 2026
@andresgur we apologize for your frustration but please watch your language.
@neutrinoceros didn't we merge a PR that you wrote a while back to address this?
andresgur commentedon Jan 27, 2026
apologies @pllim it was a long and frustrating day, and I just need it to vent. I very much appreciate your work and I understand it is by no means an easy issue you are trying to solve
PD: this was in astropy 7.2.0
neutrinoceros commentedon Jan 28, 2026
very sorry to hear that @andresgur
Let me give you a bit of additional context and maybe some hope for the near future:
we spent a lot of time digging into the legacy behavior of this system as we worked on astropy 7.2.0, and as the person who championed this effort, I must say I 100% agree with you: it is extremely confusing and in need for an overhaul. Because we didn't want to immediately break anything (or anyone) that was relying on this very old system (even unknowingly, or by accident), we agreed to first introduce (what ended up being a lot of) warnings for as many of these fragile cases as we could. The result is that astropy 7.2.0 is both very noisy and annoying to work with (aka worst of both worlds). But that's only our first step towards salvation: we are actively planning to overhaul this system for a much much simpler one in astropy 8.0.0. I also got fundings to work specifically on this, so there's real guarantees that it will happen. See #18964
andresgur commentedon Jan 28, 2026
great to hear that @neutrinoceros , keep up the good work 👍