Repository navigation
Non-deterministic endpoint selection issue #3747
Description
Activity
Lets consider a catalog having endpoint for both
shareandsharev2endpoints. When callingNewSharedFileSystemV2it callsinitClientOpts(client, eo, "shared-file-system", 2)which adds the slice{"sharev2", "share"}in endpoint opts aliases through endpoint_search.go.This is correct.
Then in endpoint_location.go
if (slices.Contains(opts.Types(), entry.Type))when fetching endpoint, it will go over the catalog and select any endpoint that matches any of the service within the Aliases endpoint opts.And so is this...except we don't use the
V3EndpointURLyou've linked anymore 🙂 (if you look a little higher in that file, you'll find a note to remove it). That means this:As a result, if
sharehappens to be processed first, this endpoint will be selected instead of thesharev2one assumed by callingNewSharedFileSystemV2.is not correct (or shouldn't be correct).
The commits you've linked are part of a larger PR, #3351. If you look through that PR, you'll note we reworked a few things but the chief one that you probably care about is the change of the default
EndpointLocatorused for our clients. If you create your client withAuthenticateorAuthenticateV3fromopenstack/client.go, theEndpointLocatorimplementation there now defaults toV3Endpointinstead ofV3EndpointURL.V3Endpointiterates through the combination of service types (built from the aforementionedEndpointOpts.TypeandEndpoint.Aliases) and use the new (as of that PR)endpointSupportsVersionhelper to determine which of them actually supports your requested version.I suspect you are creating clients via a different manner that perhaps uses the old, deprecated
V3EndpointURLfunction which is not version aware. Could you check if you're able to reproduce this with the minimal snippets in the README? If so, could you share the snippet of the service catalog you are seeing this issue against (extracted from the token) along with the service discovery documents for Manila and Cinder, where you seem to be having the issue. Feel free to scrub the URLs and any other identifying information.Hello @stephenfin !
Thanks for your quick and detailed reply !
Your analysis is correct, I was using the
V3EndpointURLand not theV3Endpoint. I tested withV3Endpointand it works fine, but it bothers me that it requires an extra API call to determine the API version during endpoint selection. It creates an overhead of at least one API call in best scenario in all my workflows. And It even makes this call when there is no ambiguity in catalog.I really do not want to add this overhead in my code. I am considering writing a custom
EndpointLocatorto implement a custom mechanism that would not require an extra API call.Have you even had any feedbacks about this version compatibility overhead ?
Thanks !
Hello @stephenfin !
Thanks for your quick and detailed reply !
Your analysis is correct, I was using the
V3EndpointURLand not theV3Endpoint. I tested withV3Endpointand it works fine, but it bothers me that it requires an extra API call to determine the API version during endpoint selection. It creates an overhead of at least one API call in best scenario in all my workflows. And It even makes this call when there is no ambiguity in catalog.This is how version discovery is and always has been intended to work. The whole flow was well documented by the API SIG some years ago. Keystoneauth has always done this correctly but Gophercloud wasn't until the series of PRs culminating in #3351. Gophercloud happened to work previously because DevStack was deploying services with multiple API versions using version suffixed aliases (
volumev3,sharev2) and Gophercloud had hardcoded preferences for this. As soon as it started using the correct service types, this gap was exposed.I really do not want to add this overhead in my code. I am considering writing a custom
EndpointLocatorto implement a custom mechanism that would not require an extra API call.The
EndpointLocatoris pluggable to allow exactly this. You are also free to fork, of course.Have you even had any feedbacks about this version compatibility overhead ?
Just here, that I'm aware of. But again, we are following spec.
Thanks for your explanations about version discovery, I managed to avoid this behavior with a custom
EndpointLocatorthat better fits our OpenStack deployment. I reckon that following the specs is indeed the right path, and I thank you for keeping the SDK flexible enough to allow us to tweak it to our needs.Thanks for your help and contributions to this project ! Have a good day !
What happened?
Hello,
I recently moved our codebase from gophercloud v1 to gophercloud v2 (yes, it's about time). I noticed a regression regarding endpoint selection in catalog.
In short, the new aliases mechanism for endpoint selection causes a non-deterministic endpoint selection from catalog. For example, if a catalog contains endpoints for both
shareandsharev2endpoints, usingNewSharedFileSystemV2can arbitrary select either endpoint from catalog in a non-deterministic way.This behavior has been introduced in several steps:
Lets consider a catalog having endpoint for both
shareandsharev2endpoints. When callingNewSharedFileSystemV2it callsinitClientOpts(client, eo, "shared-file-system", 2)which adds the slice{"sharev2", "share"}in endpoint opts aliases through endpoint_search.go.Then in endpoint_location.go
if (slices.Contains(opts.Types(), entry.Type))when fetching endpoint, it will go over the catalog and select any endpoint that matches any of the service within the Aliases endpoint opts. As a result, ifsharehappens to be processed first, this endpoint will be selected instead of thesharev2one assumed by callingNewSharedFileSystemV2.The same behavior applies to any service having multiple aliases, like block-storage for example.
Moreover, this feature that seems to check that the version is correctly enforced does not prevent this behavior at all since it does no check the service version in catalog but only in client provided filters. You can still get a
volumeendpoint when callingNewBlockStorageV3.Currently, I reverted some of these changes in my fork to be able to move forward, but I would like very much to discuss about an actual fix to this weird behavior. Specifically with @stephenfin that seems to have done lots of work on these refacto (thanks ny the way for all this work ❤️ ).
Thanks !
Version
v2 (current)