Tracking issue: future evolutions in cache and config discovery strategy #19803

I just landed a significant overhaul of the underlying mechanism in #19575

In short, phase 1 (see after) added new APIs (temporary_(cache|config)_dir_path) that are meant to replace legacy ones (set_temp_(cache|config)). In order to preserve maintainability, we should aim to drop legacy ones, but because new ones are not drop-in replacements, we also need to spread the friction over a long deprecation cycle rather than make sudden breaks if we can avoid them.

This issue, will serve as a tracker for long term planning around this evolution, including a retrospective for context.
It's also legacy sequel to #18964.

phase 0 (done, landed in 7.2.0)

• UX: warn when env variables are ignored in cache and config directory discovery #17934

phase 1 (done, expected to land in 8.0.0)

• ENH: overhaul cache and config dir discovery #19575
• API: avoid creating cache directories on discovery #19631
• ENH/BUG: avoid test pollution from data tests by making APIs resilient against missing cache directories #19650

As of phase 1, legacy APIs are "soft deprecated": no longer recommended for new code, but not formally deprecated yet, so they remain usable everywhere that new APIs can't replace them yet (if any).

phase 2 (target: 8.1.0 or 9.0.0)

• coordidnate with known downstream users and help them migrate if needed
• collect feedback on the migration path
• migrate internal usage of legacy APIs to new ones

phase 3 (target: phase 2 + >=1 major bump)

• API: Deprecate get_config_dir and get_cache_dir #19609

phase 4 (target: phase 3 + >= 1major bump)

remove legacy APIs
This should be done as late as possible. There is no rush at the time of writing.