Use pooch to manage package data and cache #17566

Activity

1. pllim

   added

   Feature Request

   utils

   Package-expert

   Effort-high

   on Dec 19, 2024
2. nstarman commented on Dec 23, 2024

   

   nstarman

   on Dec 23, 2024

   Member

   More actions

   I took a quick look at our API to see what parts of the backend pooch could replace.
   TL;DR. The actual act of downloading looks to be pretty easily replaced by pooch.retrieve which unifies all our disparate downloading code into a single function. We can push further, using a pooch.Pooch data manager. We probably want to do this for Astropy data management internally, but in these general purpose downloader functions, 🤷.

   Non-goal: remove these functions and make people learn a new API.
   Goals:

   1. replace the backend with something under active development and use, meaning if/when something breaks astropy devs have a community that's probably already working on the problem and don't need to themselves grok ftp minutiae. 🤝
   2. Document the backend to users, keeping our current frontend, so they can benefit from the additional features. ⚡
   3. Upstream useful stuff, thereby moving it out of our busy hands. ⬆️=😄
   4. Possibly move this sub-package out of utils, since it has use outside of Astropy. 🌐

   key: ⬜: N/A to pooch. ⬆️: missing from / upstream to pooch. ⬇️ : use pooch as backend

   Astropy API	Pooch	Comments
   CacheDamaged	⬆️	pooch should expose its errors / warnings. It's not clear if this specific error should be upstreamed.
   CacheMissingWarning	⬆️	pooch should expose its errors / warnings. It's not clear if this specific error should be upstreamed.
   Conf	⬇️⬜	use this to control creation of pooch.Pooch
   cache_contents	⬜	pooch deals with downloading data, not the cache once downloaded
   cache_total_size	⬜	N/A to pooch
   check_download_cache	⬆️	this seems useful to upstream, for pooch to check a cache given the hashes
   check_free_space_in_dir	⬜	N/A to pooch
   clear_download_cache	❔	pooch has a "fetch" method, but maybe a "bury" would be useful to add
   compute_hash	⬇️	pooch.file_hash
   conf	⬇️⬜	use this to control creation of pooch.Pooch
   download_file	⬇️	call to pooch.Pooch.fetch or pooch.retrieve, depending on context
   download_files_in_parallel	❔	don't need to change, since calls download_file, except should use tqdm. Also opportunity to upstream, e.g. a 'retrieve_many' or 'fetch_many' or a context-manager for customizing the downloader where we write a parallel downloader, following link
   export_download_cache	⬜	N/A to pooch
   get_cached_urls	⬜	N/A to pooch
   get_file_contents	⬜	N/A to pooch
   get_pkg_data_contents	⬜	N/A to pooch
   get_pkg_data_filename	⬇️	this is the output of pooch.retrieve or pooch.Pooch.fetch
   get_pkg_data_filenames	⬜	N/A to pooch
   get_pkg_data_fileobj	⬜	N/A to pooch
   get_pkg_data_fileobjs	⬜	N/A to pooch
   get_pkg_data_path	⬜	N/A to pooch
   get_readable_fileobj	⬜	N/A to pooch
   import_download_cache	⬜	N/A to pooch
   import_file_to_cache	⬜	N/A to pooch
   is_url	⬜	N/A to pooch
   is_url_in_cache	⬜	N/A to pooch

   🚀React with 🚀1Reacted by Clément Robert
3. nstarman commented on Dec 23, 2024

   

   nstarman

   on Dec 23, 2024

   Last edited by nstarman

   Member

   More actions

   @leouieda Thanks for making this great package! We are looking at pooch to possibly replace the downloading parts of astropy's data backend. On the above list of our API and how it relates to pooch I identified (marked with ⬆️) a few things which might be useful to upstream. I was just wondering what you thought of these: publicly exposing warnings / errors related to the target directory, and a function to check the health of a directory.
4. timj commented on Mar 18, 2025

   

   timj

   on Mar 18, 2025

   Contributor

   More actions

   Does pooch support cache expiration? I'm not sure it does from the docs.

   Also, we use signed URLs a lot at Rubin and they tend to cause havoc with URL resource caching.

   👍React with 👍1Reacted by Nathaniel Starkman