Repository navigation
skopeo inspect: a way how to avoid fetching all tags from repository #785
Description
Hello,
in our use cases we need to get only information about image, like labels, but we don't need all repoTags. We have repositories with many tags (1700+) and it is resource consuming to get all tags from registry and just drop that.
Could you please provide a way how to avoid fetching repotags?
Thank you
Activity
vrothberg commentedon Sep 12, 2019
Thanks for opening the issue, @MartinBasti!
@baude @rhatdan, that may be something for the next sprint?
rcerven commentedon Sep 13, 2019
FYI another way how to fetch all tags from quay is also :
curl -sLk https://quay.io/v1/repositories/organization/repository/tags
that does return all tags without pagination and seems like in only 1 request
rhatdan commentedon Sep 13, 2019
@vrothberg Sure.
@rcerven Do you know if this works on other registries? Artifactory? Docker.io?
rcerven commentedon Sep 13, 2019
@rhatdan it is probably specific for quay.io
main reason for us (@MartinBasti) filing this story is that when we are using skopeo inspect with quay.io
and we have a lot of tags in repositories, quay actually returns 429 error, because they are limiting number of requests per second
twaugh commentedon Sep 17, 2019
I guess it would work for any registry that automatically down-converts v2 images to v1 images, right? That's just using the Docker Registry HTTP API V1.
twaugh commentedon Oct 3, 2019
Any plan to work on this issue?
vrothberg commentedon Oct 3, 2019
We have some high priority tasks on our plate at the moment. If the issue is blocking you in some way, we can escalate it to a higher priority but I can't give an ETA.
Contributions to c/image and c/skopeo are always welcome :)
twaugh commentedon Oct 3, 2019
Thanks.
nickboldt commentedon Oct 21, 2019
Similar request in the opposite direction... I'm trying to query repos for the latest actual tag (not
:latest) but if there are a lot of tags, newer ones are not displayed, presumably because of pagination or hashing (they're not necessarily sorted logically).And for some repos, this won't ever return (too much data, times out):
curl -sLk https://registry-p.e.rh.com/v1/repositories/rh-osbs/codeready-workspaces-server-rhel8/tagsWould it be possible to add a sort/filter option to
skopeo inspectso I can ...skopeo inspect docker://registry-p.e.rh.com/rh-osbs/codeready-workspaces-server-rhel8 -c 5to return the last 5 tags?Or so I have to start parsing the value of the
.Labels.urlto get the latest actual tag, eg.,skopeo inspect docker://registry-p.e.rh.com/rh-osbs/codeready-workspaces-server-rhel8 | jq .Labels.url?mtrmac commentedon Nov 1, 2019
No, because the server API does not provide any sort/filtering facilities. Skopeo has to list all tags exactly the way
curldoes (possibly iterating through the returned pages), and it could only do filtering client-side. (As for “last 5 tags”, the tags don’t come with dates and ordering of strings is undefined/unlikely to match all the various uses.)If the server times out without returning any useful response, that must be fixed server-side (possibly adding pagination to limit the response size and time, though that might require adding extra indexes to the store, or something like that).
lcarva commentedon Sep 10, 2020
This would be very useful if we want to use skopeo to simply determine the digest of the manfiest. Currently, I don't think there's a way to retrieve this information without also retrieving all the tags in the repository (which is very inefficient for repositories with many tags).
rhatdan commentedon Oct 8, 2020
Since this can not be done based on the registry protocol, I think this issue should be close. Reopen if I am mistaken.
mtrmac commentedon Oct 8, 2020
Filtering can’t be done, not getting the data at all definitely can be done.
12 remaining items
lsm5 commentedon Sep 29, 2021
@vrothberg @mtrmac is this sufficient to close this, perhaps accompanied with a manpage update?
ktdreyer commentedon Sep 29, 2021
When I run
skopeo inspect docker://docker.io/library/python:3.9, I expect to only look at the single "3.9" tag. It works for repos with a few tags, and silently hangs for repos with thousands of tags. I'm already giving skopeo a single tag - why query pages and pages of data from the server just to throw it away? This can trigger rate-limiting on the server, etc.It's great that
--configshortcuts loading all tags, but I expect new users won't discover this easily. They'll just think skopeo is broken.mtrmac commentedon Oct 2, 2021
No; it doesn’t allow using any of the other modes.
jaikiran commentedon Oct 8, 2021
I've created #1480 to try and address this issue.
ktdreyer commentedon Oct 13, 2021
That PR #1480 uses
skopeo inspect --no-tags docker://docker.io/library/pythonas an example. In that case, the user specified no tag (so does it implylatest, likepodman pulldoes?)The issue here is slightly different.
skopeo inspect docker://docker.io/library/python:3.9ought to query that single3.9tag without trying to load all other tags.mtrmac commentedon Oct 13, 2021
AFAICS there’s no difference.
Right now, the list of tags in
RepoTagsis a list of all tags in the repository, whether or not the CLI input specified a tag (:3.9), a digest@sha256:…), or neither (defaulting to:latest); that part of the CLI input has never affected theRepoTagsvalue, but it did affect all other parts of the output (by choosing which image is inspected).The proposed
--no-tagsoption likewise disables collecting and printing theRepoTagsvalue, and doesn’t affect any other part of the input.Is there anything I’m missing?
jalbstmeijer commentedon Oct 13, 2021
I might have misread where this issue was heading too. I was hoping to get all tags back attached to the specific matching image digest.
So when an specific image has more tags associated than the one I query for, it would return these and never all tags associated to the repository.
ktdreyer commentedon Oct 13, 2021
@mtrmac Thank you for explaining that. I understand more about how
RepoTagscurrently works now - it's just a plain dump of every tag regardless of input.mtrmac commentedon Oct 13, 2021
That’s not how I read the original request, and it’s also not something the Registry API allows doing efficiently; the only way to do that is to list all tags, and then, individually one by one, fetch the tag contents to see if they point to the requested digest.
arewm commentedon Nov 17, 2021
Can this issue be closed now that #1480 is merged and
1.5.1is released?rhatdan commentedon Nov 17, 2021
SGTM