Core: DeepCopy method for the provider client

@jtopjian

@jtopjian not sure if I miss this feature, but I need to initialize a provider client, then copy it for multiple go rotines. Does gophercloud support this?

I haven't personally done that, so I'm not sure how easily it's natively supported.

What limitations are you hitting with creating a new provider client "from scratch"?

What limitations are you hitting with creating a new provider client "from scratch"?

I don't want to issue an auth logic every time I create a provider. In addition it must support reauth functionality, because I don't know how long each go routine will run.

UPD: in addition each provider must have its own context and http transport. The only shared stuff I need is auth options + token. Now sure how to achieve this.

The only shared stuff I need is auth options + token

You're looking for a way to create a client that doesn't have to hit Keystone for the initial authorization, right? So you'd provide a scoped token and begin using the client?

That makes sense. I could have swore there was a way to set a scoped token, but maybe I'm mistaken. If I get some time, I can look into this, but as an aside, getting time to work on Gophercloud has been difficult for me this year.

I managed to solve the issue by creating a custom cloneAuth function, based on v3auth.

// global client
var client *gophercloud.ProviderClient
// shared auth options
var ao *gophercloud.AuthOptions

func cloneAuth(c *gophercloud.ProviderClient, endpoint string, opts tokens3.AuthOptionsBuilder, eo gophercloud.EndpointOpts, reauth bool) error {
        // Override the generated service endpoint with the one returned by the version endpoint.
        v3Client, err := openstack.NewIdentityV3(c, eo)
        if err != nil {
                return err
        }

        if endpoint != "" {
                v3Client.Endpoint = endpoint
        }

        if reauth {
                var result tokens3.CreateResult
                switch opts.(type) {
                case *ec2tokens.AuthOptions:
                        result = ec2tokens.Create(v3Client, opts)
                case *oauth1.AuthOptions:
                        result = oauth1.Create(v3Client, opts)
                default:
                        result = tokens3.Create(v3Client, opts)
                }

                err = c.SetTokenAndAuthResult(result)
                if err != nil {
                        return err
                }

                catalog, err := result.ExtractServiceCatalog()
                if err != nil {
                        return err
                }

                c.EndpointLocator = func(opts gophercloud.EndpointOpts) (string, error) {
                        return openstack.V3EndpointURL(catalog, opts)
                }
        }

        if opts.CanReauth() {
                // here we're creating a throw-away client (tac). it's a copy of the user's provider client, but
                // with the token and reauth func zeroed out. combined with setting `AllowReauth` to `false`,
                // this should retry authentication only once
                tac := *c
                tac.SetThrowaway(true)
                tac.ReauthFunc = nil
                tac.SetTokenAndAuthResult(nil)
                var tao tokens3.AuthOptionsBuilder
                switch ot := opts.(type) {
                case *gophercloud.AuthOptions:
                        o := *ot
                        o.AllowReauth = false
                        tao = &o
                case *tokens3.AuthOptions:
                        o := *ot
                        o.AllowReauth = false
                        tao = &o
                case *ec2tokens.AuthOptions:
                        o := *ot
                        o.AllowReauth = false
                        tao = &o
                case *oauth1.AuthOptions:
                        o := *ot
                        o.AllowReauth = false
                        tao = &o
                default:
                        tao = opts
                }
                c.ReauthFunc = func() error {
                        err := cloneAuth(&tac, endpoint, tao, eo, true)
                        if err != nil {
                                return err
                        }
                        c.CopyTokenFrom(&tac)

                        // update the token for the source provider
                        lock.Lock()
                        client.CopyTokenFrom(&tac)
                        lock.Unlock()

                        return nil
                }
        }

        return nil
}

func cloneProvider(requestID string) (*gophercloud.ProviderClient, *DefaultLogger, error) {
        // initialize a provider client
        provider, err := openstack.NewClient(client.IdentityEndpoint)
        if err != nil {
                return nil, nil, fmt.Errorf("failed to create an OpenStack service client: %s", err)
        }

        var logger *DefaultLogger
        if debug {
                logger = &DefaultLogger{requestID}
                provider.HTTPClient = http.Client{
                        Transport: &RoundTripper{
                                Rt:     &http.Transport{},
                                Logger: logger,
                        },
                }
        }

        provider.TokenID = client.TokenID
        provider.EndpointLocator = client.EndpointLocator
        err = cloneAuth(provider, "", ao, gophercloud.EndpointOpts{}, false)
        if err != nil {
                return nil, nil, fmt.Errorf("failed to auth an OpenStack service client: %s", err)
        }

        return provider, logger, nil
}

func someFunc() {
        ... ... ...
        // clone a provider client
        provider, logger, err := cloneProvider(requestID)
        if err != nil {
                return nil, err
        }
        ... ... ...
}

@kayrus

@kayrus Thanks for this. If I'm reading this correctly, the key is the global client variable that contains a token which will be copied to additional cloned clients?

@jtopjian

@jtopjian correct.

@kayrus

@kayrus Understood.

I don't think there's an opportunity to turn this into a utility function, do you? If not, do you think this would be useful as documentation or should we just close this and leave it for people to find as reference?

@jtopjian

@jtopjian now I'm thinking that it would be better to have a dedicated log prefix (requestID) through the context. Unfortunately you still need to clone the provider, because Context is per gophercloud.ProviderClient, not per action/request.

Introducing a number of new functions with the WithContext suffix is too much. Do you have an idea on how to implement a context support with a low effort?

I have the same problem here. I want to integrate tracing in my project and find I can't inject context to gophercloud's outgoing request.

Introducing a number of new functions with the WithContext suffix is too much

I would like to be able to accept a per-call context.Context.

One backwards-compatible (and low-effort) implementation could be to implement a new WithContext(ctx context.Context) method on ServiceClient that returns a copy of the ServiceClient with a Context override.

The API would be:

	ctx := context.Background()
	allPages, err := servers.List(client.WithContext(ctx), servers.ListOpts{}).AllPages()

I am creating PR #2274 as a proof-of-concept, should anybody have the time to review it.

Just to make sure it's asked and a lot of in-depth architectural changes aren't made, what is the use-case that this will solve that can't be solved by a RoundTripper?

I think the concept of "tracing" is different than the need for a persistent token that was originally discussed in this issue?

The difference is when user can inject the context (or provide a RoundTripper). For now, it can only be set when you init the gophercloud.ProviderClient, not each API call.

For example, I'm developing a web service that will call OpenStack API. I create a global gophercloud.ProviderClient, and then in my own http handler, I reuse that client to access OpenStack API. I want to pass the context from my own handler to each outgoing http client, not a global client.

The origin issue seems a workaround when context can't be set on service client. So want to clone the provider client and reset the context for it.

The difference is when user can inject the context (or provide a RoundTripper)

So can a RoundTripper be used? If so, that is entirely possible to do right now.

I want to pass the context from my own handler to each outgoing http client, not a global client.

What kind of context are we talking about? If you had the ability to pass context to every Gophercloud-based API call, what kind of information would be beneficial that can't already be either 1) handled in a RoundTripper or 2) added as more of a global feature in Gophercloud?

The reason I'm pressing for details is because I want to determine if:

You folks are actually looking to implement a single use-case that can be done in a much more simple form than making large changes across Gophercloud.
There are so many varied use-cases that are different from Gophercloud function to function that it warrants each function having its own context parameter.

Using this example:

allPages, err := servers.List(client.WithContext(ctx), servers.ListOpts{}).AllPages()

This implies that the servers.List function will be doing something with ctx. Likewise, if we have:

allPages, err := images.List(client.WithContext(ctx), nil).AllPages()

Then images.List will also be doing something with ctx. Will they be doing different things? Or are you just trying to get a ctx into the service client? If the former, I'd like to learn more about these varied use-cases. If the latter, look at making a change more globally than to each function call.

err - my copying and pasting was incorrect and I pulled the wrong example. I was focusing on Alternative B being discussed in #2274:

allPages, err := servers.List(client, servers.ListOpts{Context: ctx}).AllPages()

But the core of my message is still the same: I'd like to know if everyone is asking for the same single use-case in a different way or if each function in Gophercloud truly needs to handle context in a unique way.

To complete my use case(tracing):

I want to init the provider client with custom RoundTripper first.

provider.HTTPClient.Transport = otelhttp.NewTransport(http.DefaultTransport)

The on each request, I'd like to pass the tracer context.

ctx, span := tracer.Start(ctx, "Example")
defer span.End()

servers.List(client.WithContext(ctx), servers.ListOpts{}).AllPages()

Understood - thanks!

I think there are a number of ideas happening in this issue (and in #2274) and it might be worth isolating them for simplicity.

Persistent/Shared token for authentication. This was the original reason why this issue was opened. Would a service-level context solve this? If not, persistent/shared token is a legitimate use-case that needs to be addressed in some way.
Service-level context. Similar to adding a Provider-level context, a service-level context does make sense. However, only within the same amount of detail as the provider-level context was added. If all that is needed to make tracing tools work is something like WithContext or setting client.Context, then the implementation should be no more complicated than support context.Context for outgoing HTTP requests #1480
It feels like the example give in Alternative B of service_client: Accept per-call Context #2274 implies that there is more of a need for Gophercloud to enact on a context rather than just support a context field. Is this true? And if so, what are the use-cases for different functions to pass in an actual context parameter rather than have the client struct with a Context field?

As always, I realize I may be missing a big piece of information and could be totally incorrect with the above. I do apologize if that is the case.

We should be able to close this with Gophercloud v2 out.

Reopening. We still want to be able to deep-copy provider and service clients, particularly in light of the new endpoint discovery functionality.

Navigation Menu

Repository navigation

Core: DeepCopy method for the provider client #1987

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Core: DeepCopy method for the provider client #1987

Description

Activity

jtopjian commented on Jun 23, 2020

kayrus commented on Jun 23, 2020

jtopjian commented on Jun 23, 2020

kayrus commented on Jul 15, 2020

jtopjian commented on Jul 21, 2020

kayrus commented on Jul 22, 2020

jtopjian commented on Jul 30, 2020

kayrus commented on Sep 2, 2020

zhsj commented on Nov 11, 2021

pierreprinetti commented on Nov 11, 2021

jtopjian commented on Nov 12, 2021

zhsj commented on Nov 12, 2021

jtopjian commented on Nov 12, 2021

jtopjian commented on Nov 12, 2021

zhsj commented on Nov 12, 2021

jtopjian commented on Nov 12, 2021

pierreprinetti commented on Jul 16, 2025

stephenfin commented on Aug 7, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions