On Microversions

This issue is an attempt to collect and organize different topics related to microversions which have been taking place in other PRs and Issues.

This issue is to discuss high-level implementation decisions, not about how the decision will actually be implemented. Please refrain from discussing the performance benefit of strings.Contains vs regexp.MatchString or how a specific implementation saves n additional function calls.

Background Docs:

External examples:

Microversion Logs:

Topics raised so far:

To what extent does Gophercloud need to be concerned about microversions?

Using the above hypervisors.py link as an example, if Gophercloud were to match functionality, then microversions will play a significant role. The above link is useful for illustrative purposes because the file is small yet several microversion-specific features are used:

"guarding" API calls with a min and max microversion
Altering functionality in the midst of an API call based on the microversion.

In addition, an example has been cited which shows additional Fields being added in new microversions. This may have an effect on Gophercloud's result structures.

But why is there so much microversion logic in the client?

How will API versions be retrievable via Gophercloud?

Decision: Each service will have a versionless apiversions package.

How will Gophercloud send a microversion to the API service?

Decision: Support for sending the service-specific version header was added in 5acaa3c.

What microversion information should the ServiceClient hold?

The ServiceClient currently supports setting and retrieving a single microversion value at client.Microversion.

A question of whether the ServiceClient should also store minimum and maximum microversion of the API service has been raised. python-novaclient only seems to be concerned about a single value: the microversion set in the client. That's a good precedence to go off of, but I'll leave this item open for discussion.

Should Gophercloud automatically populate the ServiceClient with a requested microversion, the API service's minimum supported microversion, or nothing at all?

Please see the points made in the spec under "Clients should expect the following behavior from the server".

My interpretation of the above points is that the API service should be quite lenient toward the client not specifying a microversion at all. Problems may arise when the API service has not implemented functionality to make those points work. The client may then see unexpected behavior.

Should Gophercloud prevent a client from calling a method if client.Microversion is incompatible?

I might be incorrect about this, but it seems as though the only reason for doing this is to provide the user with an error that the API service itself has chosen not to respond with. According to the spec, if the client was allowed to make any API call, the service would respond with a clear error the client's currently set microversion is not compatible as well as which microversion(s) are compatible.

Referring to a statement earlier ("But why is there so much microversion logic in the client?"), having the client handle a large amount of microversion functionality might be to provide a better end-user experience when the API service has chosen to use microversions, but has not implemented the correct 406 response.

Given that I've found examples of preventing/guarding in three different Python clients, either this is standard practice or there's another reason for limiting so many calls with the compatible microversions that I'm not aware of yet.

Should Gophercloud alter result struct fields based on the microversion?

I think this just made #300 even more difficult.

Should Gophercloud alter behavior of an API call depending on the microversion?

There's already precedence of Gophercloud implementing extra logic in building some API calls. Altering behavior based on microversion seems no different.

@jtopjian

@jtopjian Great write-up; thank you for summarizing the issue.

What microversion information should the ServiceClient hold?

The current Microversion field is intended to be populated by the user to represent the microversion their application uses. At most, I'd say maybe a minMicroversion and a maxMicroversion field (notice they're unexported), but it's unclear to me whether they'd be generally useful.

Should Gophercloud automatically populate the ServiceClient with a requested microversion, the API service's minimum supported microversion, or nothing at all?

I don't have strong feelings about this, but I think because the service client is long-lived, and in most cases should only need to be created once, perhaps fetching the max and min versions and setting them in the suggested, unexported fields above at creation time would be OK.

Should Gophercloud prevent a client from calling a method if client.Microversion is incompatible?

I'm not sure how we'd know if a method was incompatible. Would each service (or package?) have a SupportedMicroversion global variable? Or would the ServiceClient have a field for this that gets populated on creation? Would such a variable need to be bumped every time a feature was added to the service?

Should Gophercloud alter result struct fields based on the microversion?
Should Gophercloud alter behavior of an API call depending on the microversion?

I think these two questions are closely related. I'll need to think about this a bit to see how such a design may look. We already (would like to) modify results based on extensions, but as you say, this makes even that more complicated.

Topic: What microversion information should the ServiceClient hold?

One benefit of storing min and max would be to do the following:

err := client.SetMicroversion("2.88")
// err: 2.88 is outside of the range of min and max

Where SetMicroversion is a method that compares the user's input with the discovered min and max.

I think this is helpful and useful and I wouldn't oppose it.

Topic: Should Gophercloud automatically populate the ServiceClient with a requested microversion, the API service's minimum supported microversion, or nothing at all?

I think having microversion information available at the time the client is created is certainly helpful and given how many features are now mixed up in microversions, it will save users/developers an extra step of creating a client and then setting a microversion.

However, I'm not sure what to use as a default setting.

python-novaclient hard-codes the default version: https://github.com/openstack/python-novaclient/blob/master/novaclient/__init__.py#L28

Since gophercloud does not cut releases, this isn't realistic (nor maintainable).

So, what about automatically setting a sane default value? The spec says a min_version and max_version should be published. However, at the moment, Nova is only publishing min_version and version (go figure).

I'm leaning on the following priorities:

if max_version != "" {
  ...
} else if version != "" {
  ...
} else if min_version != "" {
  ...
}

IMO, max_version should be safe to use if the catalog is publishing it. I might be completely wrong about that, though. With my operator-hat on, it makes me a little nervous. However, ensuring Gophercloud-based apps can easily take advantage of new features, it makes sense.

Providing options to client creation might get too embellished. For example, adding something like a Microversion field to EndpointOpts where valid values are max_version, min_version, version, latest (keyword), or a specific version.

Topic: Should Gophercloud prevent a client from calling a method if client.Microversion is incompatible?

I'm not sure how we'd know if a method was incompatible. Would each service (or package?) have a SupportedMicroversion global variable?

Something like that. Take a look at the python client links. Some methods have the following:

@api_versions.wraps("2.0", "2.32")
...
@api_versions.wraps("2.33")

This will restrict the method to only being valid when the client's microversion is between 2.0 and 2.32. As a test, I increased the version to a high erroneous number and got:

$ nova hypervisor-list
ERROR (VersionNotFoundForAPIMethod): API version '2.41' is not supported on 'novaclient.v2.hypervisors.[class HypervisorManager(base.ManagerWithFind):].list' method.

Where 2.41 is the default version hard-coded in the version of python-novaclient I'm using.

Running nova --debug ..., I can see that no attempt was ever made to contact the Nova API service. The call was blocked at the client.

@jtopjian

@jtopjian Thanks a lot for putting all the information together.

Topic: Should Gophercloud prevent a client from calling a method if client.Microversion is incompatible?
Topic: Should Gophercloud alter result struct fields based on the microversion?
Topic: Should Gophercloud alter behavior of an API call depending on the microversion?
For me all the above topics are related.

In my understanding, microversions are means of introducing new features without having to increase the API major version. Increasing major API version is costly so instead of it there is probably a cheaper way: microversion, i.e. increase of API minor version.

How are microversions implemented on the server side? Pieces of code are guarded by specific microversion interval checks like @api_versions.wraps("2.0", "2.32").

What is the result for the client side? Different server API microversions behave in different way. The differences are not big, however, the differences exist.

How does gophercloud handle different API major versions? gophercloud have separate code in separate directories for different API major versions. So the gophercloud code is guarded by specific API major version.

Should gophercloud alter behaviour depending on microversion? Yes, as the server-side pieces of code are guarded by specific microversion interval the same must be done on the client-side, i.e. in gophercloud.

So I answer to all the 3 "Should ..." questions: "YES".

Topic: To what extent does Gophercloud need to be concerned about microversions?

Important: every microversion represents a separate API specification. So in case gophercloud user chooses "2.11" microversion gophercloud user expects that gophercloud will behave exactly according to "2.11" specification.

I expect gophercloud to behave exactly according to particular microversion I choose. So gophercloud must be "very concerned" about microversions.

Topic: What microversion information should the ServiceClient hold?

Information whether gophercloud user switched the microversions feature on or off. Currently, I interpret client.Microversion == "" as microversions feature switched off. I'd like to have the microversions feature switched off by default to keep backwards compatibility.
User requested microversion, e.g. "2.17". I assume it's stored in client.Microversion.
Server side maxMicroversion. In case user chooses client.Microversion == "latest" it means that the user chose server-side maxMicroversion. That's why the maxMicroversion needs to be stored.
Server side minMicroversion. Currently, I don't see any use of storing server side minMicroversion except checking that the microversion chose by user is within server side minMicroversion, maxMicroversion interval. However, I more inclined to store the server side minMicroversion than not storing it.

Edit:
4. Server side minMicroversion. In case user chooses client.Microversion == "" it means that the user chose server-side minMicroversion. That's why the minMicroversion needs to be stored.

Topic: Should Gophercloud automatically populate the ServiceClient with a requested microversion, the API service's minimum supported microversion, or nothing at all?
I'd leave the microversions feature switched off (i.e. client.Microversion == "") to keep the backwards compatibility.

I'd set the server-side minMicroversion and maxMicroversion to reasonable defaults without querying the server when the ServiceClient is created. So in case ServiceClient for v2 endpoint is created I'd set both minMicroversion and maxMicroversion to "2.0".

I'd query the server for minMicroversion and maxMicroversion as late as the user switches the microversions feature on, e.g. calls client.SetMicroversion("2.88") method.

Topic: To what extent does Gophercloud need to be concerned about microversions?

Important: every microversion represents a separate API specification. So in case gophercloud user chooses "2.11" microversion gophercloud user expects that gophercloud will behave exactly according to "2.11" specification.

I expect gophercloud to behave exactly according to particular microversion I choose.

That's a really good point.

Topic: What microversion information should the ServiceClient hold?

Server side maxMicroversion. In case user chooses client.Microversion == "latest" it means that the user chose server-side maxMicroversion. That's why the maxMicroversion needs to be stored.

I don't think this is correct. If the microversion is set to "latest", the word "latest", verbatim, is sent to the server. Meaning, the following header is sent, regardless of what the API service is advertising in the catalog:

X-OpenStack-Nova-API-Version: latest

There's no need to track maxMicroversion in order to do this.

Topic: Should Gophercloud automatically populate the ServiceClient with a requested microversion, the API service's minimum supported microversion, or nothing at all?

I'd leave the microversions feature switched off (i.e. client.Microversion == "") to keep the backwards compatibility.

A few notes about this:

If no microversion is set, the server implicitly acts as if the client had sent a header with the the min_version advertised by the api service. Once an API service has adopted microversions, as soon as it begins advertising a min_version, a client will begin using microversions, whether implicitly or explicitly.

This implicitness might present problems if Gophercloud chooses to adopt the guarding method of the Python clients.

For example, a client is created, no microversion is set by default, and a client makes an API call. If Gophercloud chooses to guard API calls, the client would not be able to make otherwise valid calls. The server's min_version is set to, say, 2.5, but because client.Microversion is set to "", Gophercloud would prevent the client from running anything supporting 2.1, 2.2, etc.

The same holds true for modified result structures and altering API requests.

The Python client libraries are hard-coding a min microversion that all clients are set to. This is most likely to prevent the above example from happening.

If Gophercloud does not set a microversion by default, it is deviating from the behavior of other SDKs (which is not always bad, but needs to be thoughtfully considered), and might present problems implementing other microversion features.

With all of that in mind, if the client tracked minMicroversion, then if client.Microversion was not set, the client could fall back to minMicroversion and be able to handle all of the guarding, modifications, etc.

But I don't see a difference between tracking minMicroversion and just setting client.Microversion to min_version by default.

Maybe this is what python-novaclient et al are doing when they hard-code the default microversion... it could just be that the hard-coded version was the API service's min_version at the time of release, but since microversions change very frequently, this becomes out of sync quickly. Example: my current environment has a min_version of 2.1, but the client defaults to 2.41. It could be that the python-novaclient package installed is slightly newer than the nova-api package installed.

Topic: What microversion information should the ServiceClient hold?

Server side maxMicroversion. In case user chooses client.Microversion == "latest" it means that the user chose server-side maxMicroversion. That's why the maxMicroversion needs to be stored.

I don't think this is correct. If the microversion is set to "latest", the word "latest", verbatim, is sent to the server. ...

I'm sorry I've not described all my reasoning. I shortened it a way too much.

I've already described that every particular microversion represents a separate API specification. So in case of Manila API, there are currently many independent specifications, particularly "2.0", "2.1", "2.2", ... "2.22", ... All of them are pretty similar, but none of them are same.

How will we implement in gophercloud these pretty small differences? I assume we will uses different techniques. I assume that one of the techniques will be func CompatibleMicroversion that checks if the microversion chosen by the user is same as the specification microversion the code is for. Typically a piece of source code is for specification version "2.7+" so the func CompatibleMicroversion will check whether the user chose microversion >= "2.7".

But what if the user chooses microversion "latest"? Should the code for specification version "2.7+" be executed or not? How could we decide? IMO, the only way is to get the server side maxMicroversion and find out if the server side maxMicroversion is >= "2.7". That's why I think it's necessary to hold the server side maxMicroversion.

The same applies if the user chooses microversion "". Then the server side minMicroversion is needed for the comparison.

I came to the above while I was developing tests for func CompatibleMicroversion.

Topic: Should Gophercloud automatically populate the ServiceClient with a requested microversion, the API service's minimum supported microversion, or nothing at all?

For example, a client is created, no microversion is set by default, and a client makes an API call. If Gophercloud chooses to guard API calls, the client would not be able to make otherwise valid calls. The server's min_version is set to, say, 2.5, but because client.Microversion is set to "", Gophercloud would prevent the client from running anything supporting 2.1, 2.2, etc.

IMO, what you have described is incorrect guarding of API calls. IMO, no microversion must behave in the same way as if the server's min_version was chosen. I came across this while writing tests for func CompatibleMicroversion.

With all of that in mind, if the client tracked minMicroversion, then if client.Microversion was not set, the client could fall back to minMicroversion and be able to handle all of the guarding, modifications, etc.
But I don't see a difference between tracking minMicroversion and just setting client.Microversion to min_version by default.

That's really a good point. I agree with setting client.Microversion to min_version by default.

However, we allow gophercloud user to set the client.Microversion value, don't we? Will we consider client.Microversion == "" a valid value?

I'd set the default client.Microversion value to min_version and didn't allow user to set client.Microversion to value "". So the client.Microversion will become private client.microversion and a func client.SetMicroversion will have to be added.

Topic: What microversion information should the ServiceClient hold?
Topic: Should Gophercloud automatically populate the ServiceClient with a requested microversion, the API service's minimum supported microversion, or nothing at all?

In case I try to implement design ideas we're discussing it gives me more insight into the topic. That's why I tried to write a very early skeleton implementation of these two topics in the below commits:

Even though we do not discuss implementation I believe it may be useful.

Topic: What microversion information should the ServiceClient hold?

I apologize, but I'm not fully understanding what you're trying to convey.

However, I did a search of the python-novaclient code for the word "latest" and came across the unit tests for the API versions parsing:

https://github.com/openstack/python-novaclient/blob/22eade1b11c35c48f133ad221a8aa11fafbaab59/novaclient/tests/unit/test_api_versions.py

It's interesting that there are tests for both what the client is advertising and what the server is advertising. I don't have time right now to fully interpret what this means. I have a feeling that latest is implemented quite differently than how it is described in the spec.

Here's how I see latest used: If I were to do:

client.Microversion = "latest"

and tried to run the following:

func Foo() {
    // restrict microversions min=2.2, max=2.3
}

func Bar() {
    // restrict microversions min=2.2
}

Then calling Foo would fail and Bar would work. My reasoning behind that is because I think it's illogical that 2.3 is the latest microversion. If a function is capped with a max value, that should implicitly say that it's a previous microversion, not the latest. Otherwise, we would need to provide caps on all functions.

However, from reading the python-novaclient unit tests, I see that there are tests comparing both a client with min and max as well as a server with min and max. I believe this is more about novaclient's hard-coded min and max values. I haven't had time to fully interpret what that means for Gophercloud.

Even though we do not discuss implementation I believe it may be useful.

To be frank, what would be much more useful is researching existing microversion implementations and discussing if they are directly transferrable or not. If they are, then they can quickly be added to Gophercloud. If they are not, then they require careful consideration which may take several weeks.

@jtopjian

To be frank, what would be much more useful is researching existing microversion implementations and discussing if they are directly transferrable or not. If they are, then they can quickly be added to Gophercloud. If they are not, then they require careful consideration which may take several weeks.

@jtopjian thank you for being frank.

Topic: What microversion information should the ServiceClient hold?
Subtopic: Existing microversion implementations in existing clients.
I had a look at manila client:

root@ac4e1eaf34c7:/# dpkg -s python-manilaclient
Package: python-manilaclient
Status: install ok installed
...
Version: 1.11.0-0ubuntu1

This client doesn't support the keyword "latest" as a microversion:

root@ac4e1eaf34c7:/# export OS_SHARE_API_VERSION="latest"
root@ac4e1eaf34c7:/# manila list
ERROR: Invalid format of client version 'latest'. Expected format 'X.Y', where X is a major part and Y is a minor part of version.

The client has hard-coded client's min and max microversions:

root@ac4e1eaf34c7:/# export OS_SHARE_API_VERSION="5.5"
root@ac4e1eaf34c7:/# manila list
/usr/lib/python2.7/dist-packages/manilaclient/api_versions.py:196: UserWarning: Invalid client version '5.5'. Current version range is '2.0' through  '2.22'
  warnings.warn(msg)
+--------------------------------------+------------+------+-------------+-----------+-----------+-----------------+--------------------------------------------------------+-------------------+
| ID                                   | Name       | Size | Share Proto | Status    | Is Public | Share Type Name | Host                                                   | Availability Zone |
+--------------------------------------+------------+------+-------------+-----------+-----------+-----------------+--------------------------------------------------------+-------------------+
| 65f01d47-785f-4f98-b91f-348ccd85a907 | testmanila | 1    | CEPHFS      | available | False     | default         | overcloud-controller-2.localdomain@cephfsnative#cephfs | nova              |
+--------------------------------------+------------+------+-------------+-----------+-----------+-----------------+--------------------------------------------------------+-------------------+

In case the user chooses a microversion (OS_SHARE_API_VERSION) higher than the client's hard-coded max microversion the manila client uses in such case the client's hard-coded max microversion (seen in Wireshark in X-Openstack-Manila-Api-Version: 2.22\r\n).

In case the user chooses a microversion within hard-coded client's min max range:

root@ac4e1eaf34c7:/# export OS_SHARE_API_VERSION="2.10"
root@ac4e1eaf34c7:/# manila list
+--------------------------------------+------------+------+-------------+-----------+-----------+-----------------+--------------------------------------------------------+-------------------+
| ID                                   | Name       | Size | Share Proto | Status    | Is Public | Share Type Name | Host                                                   | Availability Zone |
+--------------------------------------+------------+------+-------------+-----------+-----------+-----------------+--------------------------------------------------------+-------------------+
| 65f01d47-785f-4f98-b91f-348ccd85a907 | testmanila | 1    | CEPHFS      | available | False     | default         | overcloud-controller-2.localdomain@cephfsnative#cephfs | nova              |
+--------------------------------------+------------+------+-------------+-----------+-----------+-----------------+--------------------------------------------------------+-------------------+

I see X-Openstack-Manila-Api-Version: 2.22\r\n in the first request that requests list of versions supported by the server. However, in the following request contains X-Openstack-Manila-Api-Version: 2.10\r\n as requested by the user.

In case the user doesn't specify any microversion (unset OS_SHARE_API_VERSION) the client uses the client's hard-coded max microversion.

Note: the server's min_version is "2.0" and version is "2.22".

How do I interpret the fact that the client has hard-coded min max microversions? I guestimate that the manila client fully implements specifications "2.0", "2.1", "2.2", ... "2.22".

However, AFAIK, gophercloud does not implement full specification, it implements only a part of a specification depending on contributions. What implications does this have? Should we always carefully consider the upper bound microversion for guarding functions with a microversion range during code reviews?

The manila client chooses the microversion used in the X-Openstack-Manila-Api-Version:

In case the user chose a microversion (OS_SHARE_API_VERSION is set) it's checked whether it is within the client's hard-coded min max microversion range and afterwards it's checked whether it is within the server's min max microversion range.
In case the user didn't choose a microversion (OS_SHARE_API_VERSION is not set), i.e. client's max hard-coded microversion is used and it is checked whether it is within the server's min max microversion range.

@jtopjian

Topic: What microversion information should the ServiceClient hold?
Subtopic: Should gophercloud have client's max microversion hard-coded as manila client?

Firstly, assume the gophercloud doesn't have any client's max microversion hard-coded. This means that gophercloud states that it's compliant with all specifications (e.g. all "2.0+") even with specifications that do not exist yet. So in case user requests high enough microversion, e.g. "2.100" and server supports such microversion there is quite a big risk that gophercloud will probaly misbehave for requests that had some incompatabilities introduced in "recent" (let's say in "2.90-2.99") microversions. In addition, gophercloud user doesn't know for which microversions gophercloud works correctly and for which microversions doesn't.

Secondly, assume the gophercloud has a client's max microversion hard-coded. This means that gophercloud states range of microversions it supports as a client. However, this limits gophercloud users to certain maximum microversion.

IMO, gophercloud shall have client's maximum microversion hard-coded.

@jtopjian please, what do you think?

Topic: What microversion information should the ServiceClient hold?
Subtopic: Existing microversion implementations in existing clients.

Python nova client has client's min max microversions hardcoded.
I tried whether python nova client supports the "latest" keyword:

root@ac4e1eaf34c7:/# export OS_COMPUTE_API_VERSION="latest"
root@ac4e1eaf34c7:/# nova list
ERROR (UnsupportedVersion): Invalid format of client version 'latest'. Expected format 'X.Y', where X is a major part and Y is a minor part of version.
root@ac4e1eaf34c7:/# export OS_COMPUTE_API_VERSION="5.5"
root@ac4e1eaf34c7:/# nova list
ERROR (UnsupportedVersion): Invalid client version '5.5'. Major part should be '2'
root@ac4e1eaf34c7:/# export OS_COMPUTE_API_VERSION="2.500"
root@ac4e1eaf34c7:/# nova list
ERROR (CommandError): The specified version isn't supported by client. The valid version range is '2.1' to '2.37'
root@ac4e1eaf34c7:/# export OS_COMPUTE_API_VERSION="2.latest"
root@ac4e1eaf34c7:/# nova list
+--------------------------------------+-------------------+--------+------------+-------------+-----------------------------------------+
| ID                                   | Name              | Status | Task State | Power State | Networks                                |
+--------------------------------------+-------------------+--------+------------+-------------+-----------------------------------------+
| 321cd374-a3f2-48e7-92a2-84030220061f | jhou_ocp36_master | ACTIVE | -          | Running     | kube_priv_net=172.16.5.7, 10.19.114.191 |
| ba3cca07-c735-40f0-be31-8c8a5229a68e | jhou_ocp36_node1  | ACTIVE | -          | Running     | kube_priv_net=172.16.5.6, 10.19.114.214 |
+--------------------------------------+-------------------+--------+------------+-------------+-----------------------------------------+

Nova client documentation says:

--os-compute-api-version <compute-api-ver>
                                Accepts X, X.Y (where X is major and Y is
                                minor part) or "X.latest", defaults to
                                env[OS_COMPUTE_API_VERSION].

So the keyword "latest" is not supported, only "X.latest" is supported.
When I used OS_COMPUTE_API_VERSION="2.latest" there is no "latest" in X-OpenStack-Nova-API-Version: 2.37\r\n, but the client's max microversion is used (note: server responded that it supports microversions from "2.1" till "2.38").

That's why I think "X.latest" in the OS_COMPUTE_API_VERSION nova client environment variable has completely different meaning than the "latest" in X-OpenStack-Nova-API-Version in the specification.
IMO, "X.latest" in the OS_COMPUTE_API_VERSION nova client environment variable is an instruction for the nova client to calculate max(client's_max_microversion, server's_max_microversion) and use the resulting particular number in the X-OpenStack-Nova-API-Version header.
While the "latest" in the X-OpenStack-Nova-API-Version header is an insturction for the server to use server's_max_microversion.

Python cinder client has also client's min max microversions hardcoded.
I also tried whether cinder client supports the "latest" keyword:

root@ac4e1eaf34c7:/# export OS_VOLUME_API_VERSION="latest"
root@ac4e1eaf34c7:/# cinder list
ERROR: Invalid format of client version 'latest'. Expected format 'X.Y', where X is a major part and Y is a minor part of version.
root@ac4e1eaf34c7:/# export OS_VOLUME_API_VERSION="2.latest"
root@ac4e1eaf34c7:/# cinder list
ERROR: The version should be explicit, not latest.
root@ac4e1eaf34c7:/# export OS_VOLUME_API_VERSION="3.500"
root@ac4e1eaf34c7:/# cinder list
ERROR: Version 3.500 is not supported by the API. Minimum is 3.0 and maximum is 3.15. (HTTP 406) (Request-ID: req-4771ff15-cef1-479c-bae3-d214c2284734)

The documentation says:

--os-volume-api-version <volume-api-ver>
                        Block Storage API version. Accepts X, X.Y (where X is
                        major and Y is minor
                        part).Default=env[OS_VOLUME_API_VERSION].

So cinder client does not support the "latest" keyword.

I check manila client for X.latest format:

root@ac4e1eaf34c7:/# export OS_SHARE_API_VERSION="2.latest"
root@ac4e1eaf34c7:/# manila list
ERROR: Invalid format of client version '2.latest'. Expected format 'X.Y', where X is a major part and Y is a minor part of version.

But manila client doesn't support this format.

To sum it up, none of the python clients (nova, cinder, manila) supports the "latest" keyword. None of them uses the "latest" keyword in the X-OpenStack-XXXX-API-Version header. So even though the specification allows the "latest" keyword in the X-OpenStack-XXXX-API-Version header none of the clients uses this possibility.

Only the python nova client uses X.latest, but the word "latest" is used for different purpose in completely different context.

@pospispa

@pospispa I've been tied up working on a few other outstanding issues. I haven't had time to fully read your comments, but I will do so soon.

I wanted to add a few notes here while they're fresh in my head:

The python-keystoneauth library recently added support for microversion and service discovery. While those two topics have two different goals, they're very closely related. There's some really good, detailed documentation here.

What's described in that document is also currently implemented in the keystoneauth code, which is good, because it might help define things that seem odd in the spec. One thing to keep in mind is that microversions are very new to OpenStack. A lot of the services haven't adopted microversions, and those that have may not have adopted all features defined in the spec (ie: the latest keyword).

Topic: Should Gophercloud automatically populate the ServiceClient with a requested microversion, the API service's minimum supported microversion, or nothing at all?

I think it makes sense to pull this information during the endpoint query being done here. I just came upon an issue where Keystone is now publishing a versionless endpoint in the catalog. You must query the version discovery URL in order to get the versioned endpoint. If that is now required, then it only makes sense to grab microversion information at the same time.

The keystoneauth doc details several endpoint options which could go in Gophercloud's EndpointOpts. I haven't investigated enough to determine if it makes sense to add all of them, though.

Topic: What microversion information should the ServiceClient hold?

IMO, gophercloud shall have client's maximum microversion hard-coded.

I really don't like the idea of hard-coding something like this, but you make very good points. I don't want to say either yes or no yet, but I recognize it as a possibility.

The more I dig into microversions, the more I do not like them.

@jtopjian

@jtopjian thank you for posting the link.
I'm looking at the keystoneauth.

@jtopjian

@jtopjian so you prefer to implement the microversions in the gophercloud library along the lines of the keystoneauth implementation.
I agree with this.

I assume that you prefer to currently implement only a part of the features the keystoneauth provides to it's users.

Topic: Should Gophercloud automatically populate the ServiceClient with a requested microversion, the API service's minimum supported microversion, or nothing at all?

I think it makes sense to pull this information during the endpoint query being done here.

IMO, this may be difficult because both func V2EndpointURL and func V3EndpointURL return (string, error) so it's difficult to extend the data type these functions return with additional microversion information.
IMO, the information can be pulled also in func initClientOpts

But anyway, we agree that it make sense to pull the microversion information during endpoint query/gophercloud.ServiceClient creation.

@jtopjian

Topic: Should Gophercloud automatically populate the ServiceClient with a requested microversion, the API service's minimum supported microversion, or nothing at all?

I just came upon an issue where Keystone is now publishing a versionless endpoint in the catalog. You must query the version discovery URL in order to get the versioned endpoint. If that is now required, then it only makes sense to grab microversion information at the same time.

@jtopjian I tried to find more information on unversioned endpoints. The only thing I found is "Service Catalog Standardization".

Unfortunately, I don't know what's the status of the "Service Catalog Standardization". Whether it's already implemented.

My understanding is that catalog can contain both unversioned and versioned endpoints. Both unversioned and versioned endpoints can be used without microversions, however, I believe it's better to firstly query the endpoint for microversion ranges. The versioned endpoint will very probably return only one microversion range while the unversioned endpoint will probably return more than one microversion range (CURRENT, SUPPORTED).

@jrperritt

I wanted to give a quick update on this. Unfortunately it's "a good solution still hasn't been identified".

Apologies if I'm speaking out of turn on @jrperritt's behalf, but he did mention that he's tried a few ideas, but they didn't pan out. So he hasn't been ignoring this issue. :)

Sometimes it's quite healthy to take a break from a difficult problem -- it's often that you'll see an entirely new way of solving it. So if anyone finds themselves in that position, definitely post your thoughts.

We do plan to continue working on a solution for this, but it might take a little longer. Having a lack of time to devote to this issue is the main reason why it hasn't been solved yet.

Hi all, i was wondering if anyone interested in reviving this discussion. Features keep coming out and everyone who depends on gophercloud (read terraform) are lagging behind.

I went through the discussion as much as i could comprehend and it seems to me that problem comes from the fact that enabling microversion may introduce new fields in request/response structs.
I would like to propose a possible solution to replace RequestOpt with an interface

type RequestOpt interface {
    Body() io.Reader
    Headers() map[string]string
}

Motivation is that ProviderClient.Request doesn't actually need to have an access to the structure and having an interface here would allow for feature specific RequestOpt implementations which would inject appropriate microversion headers and fields. This would cover for an extra request fields when microversion is enabled.

When microversion introduces new response field, feature implementation would include FeatureResponseAccessor(r http.Response) FeatureStruct function as a stand-alone
function or as a method on RequestOpt interface.

So use case would look like

type WellKnownActions string
const ActionExtend WellKnownAction = "extend"

type WellKnownFields string
const FieldSize WellKnownField

func SomeClientFunc()  {
	// Desired experience for default stuff

        // func NewRequestOpts(...OptFunc) RequestOpt  
	reqOpts := NewRequestOpts(ReqAction(ActionExtend), 
                                  ReqField(FieldSize, 4))
	resp, _ := ProviderClientRequest("POST", "/url", reqOpts)
	assertError(resp.Error())


	// Destired experience for special stuff
	reqOpts := NewRequestOpts(ReqAction(ActionExtend),
                                  ReqField(FieldSize, 4),
                                  Microversion("3.42"))
        // `Validate()`  tests if `gophercloud` supports requested microversion with requested fields.
        err := Validate(reqOpts)

        // Client being aware that he requested specific feature would call
        // an appropriate response interpreting function that comes with feature support.
        // This covers for extra fields in response introduced by microversion.
	resp, _ = ProviderClientRequest("POST", "/url", reqOpts)
	feature_resp, ok := FeatureResponseAccessor(resp)
	assertField(feature_resp.Attrs())


        // Alternatively response interpreter might be integrated in `RequestOpt` interface
        // to allow for less aware users.
        resp, _ = ProviderClientRequest("POST", "/url", reqOpts)
        interpreted_resp := reqOpts.FeatureResponseAccessor(resp)
        assertField(interpreted_resp.Attrs())
}

@flashvoid

Hi all,

Hi @flashvoid!

i was wondering if anyone interested in reviving this discussion.

Yes 😄

Personally I exhausted all ideas I had and decided to take a break. There's been more than enough non-microversion activity happening in Gophercloud that has kept me from coming back to this.

Features keep coming out and everyone who depends on gophercloud (read terraform) are lagging behind.

I went through the discussion as much as i could comprehend and it seems to me that problem comes from the fact that enabling microversion may introduce new fields in request/response structs.

Thanks for reading through everything. :)

You're correct in your analysis. The added difficulty is that we don't want to implement something that is going to require a lot of added work every time a new microversion comes out. Gophercloud is maintained entirely by volunteers so having to manage new sets of structs for each release is unmaintainable.

Using Terraform as an example is also applicable here. Whatever solution is implemented, it would be desirable to not have to have new resources for each microversion (or release).

I would like to propose a possible solution to replace RequestOpt with an interface

This is already supported, but let me know if you're thinking about using the interface differently than the following:

Any time you see a function that takes an OptsBuilder as a parameter, it's possible to extend the request structure. Again, using Terraform as an example, you can see that done here:

https://github.com/terraform-providers/terraform-provider-openstack/blob/master/openstack/types.go#L163-L173

This adds a field called ValueSpecs to the request.

(Terraform-specific side note: ValueSpecs is a special field which can add any key/value pair to the request body. Most Terraform resources support this as a way of adding vendor-specific fields. However, it can be used as a backdoor to add any field. This isn't largely advertised since we don't want to see it abused, but if a certain Terrafom resource is missing support for a certain field and the resource supports ValueSpecs, you can inject the field this way as a workaround. The caveat is that this only works for creating resources and not maintaining resources)

In addition, there are also several Gophercloud packages that implement API extensions. These are used to extend base structs:

https://github.com/gophercloud/gophercloud/blob/master/openstack/compute/v2/extensions/availabilityzones/doc.go

So in term of adding fields to requests, that part of the puzzle is the easiest.

When microversion introduces new response field

This is one of the difficult parts. The concern I have here is the user experience and ensuring the dev/user can easily distinguish (or separate) new fields and original fields.

// Client being aware that he requested specific feature would call
// an appropriate response interpreting function that comes with feature support.

Yes - this 😄

The final piece of the puzzle is handling fields which are no longer supported and might cause an error to be triggered if they were sent (even if they had no value).

What you've proposed sounds very intriguing. Would you be able to do a more detailed proof-of-concept based on an actual microversion feature? I would be more than happy to review.

I looked at this a bit more this morning and honestly I think that finding a "perfect" solution isn't going to happen. This proposal combines a number of things we've discussed here and IMO works pretty well. It's my opinion that the design of micro versions in OpenStack are going to require some compromise on the consumers part. The only part about this proposal I'm not a huge fan of is the query step, I'd rather get max supported version on init and use that rather than calling out to verify before each call.

I'm still very unsure on how we actually implement/communicate all of this when there are for example 57 versions in Cinder. Which brings us back to cherry-picking what we implement.

One last alternative I've though of is a "dumbed down client". The idea here being that we don't try and define all of these details; instead we offer an option that provides just a basic foundation:

client.GetSupportedVersions()
client.IssueRequest(reqName string, reqOptions map[string]string) // or a generic interface if preferred

reqOptions would be the standard RESTy things including header opts etc. Might even make sense to break that into two things; one being endpoint/header opts, and the second being actual requestOpts.

Note that we can pretty much already do this just using provider_client.RequestOpts, a few helpers would make it much easier but I think it's pretty doable.

The point here being that I'm not sure it's realistic to keep up with the mv's across OpenStack at this point so maybe instead of trying to keep up and defining strict interfaces we instead provide a flexible mechanism to exploit the OpenStack API's including micro-versions both now and as they continue to be added. In other words enable the use of Gophercloud as a simple language binding if desired.

So by default we'd provide the base version like we do today with all the conveniences and arg checking etc (ie training wheels), and for those that want to venture ahead provide the basics and let them implement whatever mv feature they'd like.

BTW, here's an example of a pattern I've been using that seems to work fairly "ok". I added an additional NewBlockStorage function to setup micro versions when needed, other than that I just use the raw types and extend as necessary. The following example implements volumes.create with group_id option for example:

https://gist.github.com/j-griffith/0dee9a3c871a3e6988d11be530bbd9bb

So, after reviewing CreateOptsBuilder and extensions i agree it is very similar to what i was going to propose.
There are few probably_minor advantages to passing options as an interface right into client.Request, as opposed to rendering options into map[string]interface{} and than passing that.
Pro:

remove reflect magic that happens in BuildRequestBody
one RequestOpts interface for everyone as opposed to having one interface per each opts struct (e.g. firewall.CreateOptsBuilder). (please check a demo)
much less confusing for anyone wanting to figure out how to write an extension.

Con:

changing signature of client.Request functions which affects existing code.

So, proposed demo doesn't really add much to existing capabilities in terms of microversions. My reasoning was along the lines that code won't write itself, normally, so supporting features guarded by microversion would require writing some code every time. The best that can be done here is to template that code as much as possible so that anyone wanting to contribute an extension would have to satisfy only one small interface.

I'm happy to do more proof-of-concept code and eventually contribute to refactoring if you find this a reasonable thing to do.

@j-griffith

I've spent the day recalling where I left with this and it was something along the lines of:

CreateOpts/UpdateOpts: Implement microversion tags in the fields like so:

type CreateOpts struct {
  Foo string `json:"foo" mv_min:"3.1" mv_max:"3.3"`
  Bar string `json:"bar" mv_min:"3.5"`
  Baz string `json:"baz"`
}

And then implement a Validate method or function which is called during the ToXCreateMap. Validate will check the client's microversion (which was either explicitly set by the user or (in the future) automatically detected).

Whether or not Validate throws an error or silently ignores invalid fields is up for debate.

The result structs would look identical:

type Volume struct {
  Foo string `json:"foo" mv_min:"3.1" mv_max:"3.3"`
  Bar string `json:"bar" mv_min:"3.5"`
  Baz string
}

Playing with tags in the results is the first thing I experimented with in the beginning of this discussion and learned there is no easy way to outright omit fields. But upon revisiting this, I thought about implementing something like Terraform's d.Get() system:

foo, err := volume.Get("foo")
fmt.Println(foo.(string))

AFAIK it's impossible to return foo as a string, so it needs to be casted. So that's a tradeoff.

Somehow volume would need to know about the microversion, too, so the Get method might be more like:

foo, err := volume.Get(client, "foo")

or maybe even

foo, err := client.GetField(volume, "foo")

The benefit of the above is that GetField would not have to be implemented in each resource.

One benefit to all of the above is that all microversioned fields would be alongside the base fields. No need to extend. This is a big help for maintenance.

At the time, this felt like it would be a satisfactory solution, but I never got beyond thinking about it in my head. Gophercloud began receiving a steady stream of PRs (which is awesome) and I haven't been able to come back to this.

@j-griffith

I think that finding a "perfect" solution isn't going to happen

I agree.

I'd rather get max supported version on init and use that rather than calling out to verify before each call.

This is totally possible and something I want to also get back on track with. I was doing some work previously with the apiversions packages in hopes of eventually having endpoint/version autodiscovery working.

One last alternative I've though of is a "dumbed down client".

The point here being that I'm not sure it's realistic to keep up with the mv's across OpenStack at this point

In other words enable the use of Gophercloud as a simple language binding if desired.

This is something I think about regularly. In theory, a small Go REST client that provided basic JSON interaction would work, but I feel like it would be a disservice to using Go in general. In addition, Gophercloud is definitely opinionated but I've learned to appreciate its design. It's prevented numerous bugs as well as uncovered bugs in OpenStack itself. I feel like moving away from that is, in a way, giving up from fully understanding the implementation and ensuring API interaction is done well.

However, with that said, it's still a valid feature and I would not be opposed to investigating this.

@flashvoid

Thanks for writing up the demo. I haven't looked at it in detail (I was writing this comment as it came in) yet. But to quickly address your "pros":

remove reflect magic that happens in BuildRequestBody

reflect should always be avoided when possible and params.go is most definitely not easy to understand upfront, but it I've come to appreciate it very much in one of those "if it didn't exist, it would be badly missed" type ways.

one RequestOpts interface for everyone as opposed to having one interface per each opts struct

When I first started working with Gophercloud, I thought having each resource support this amount of flexibility was overkill. But having fielded Terraform questions for 3 (!) years now, I've learned that there are so many custom modified OpenStack clouds out in the wild and this per-resource flexibility is used.

much less confusing for anyone wanting to figure out how to write an extension.

I agree with this. I'm not a fan of Gophercloud's extension system but I understand why we do it. Some day I hope to create some convenience code that presents common resources with all of their extensions (for example, a Server with everything 99% of the OpenStack clouds have defined as a Server).

With all of the above said, I am not opposed to making strong design changes to Gophercloud if it helps Gophercloud be able to keep up with the fast pace of microversions. But for non-technical reasons, these large changes might take a long time to fully vet and review -- it really comes down to a time constraint.

And one more note, we do have https://github.com/gophercloud/utils which is a much more flexible code repository. Gophercloud is opinionated and rigid and deals solely with basic API communication. utils is meant as a layer between the app and Gophercloud. Maybe a microversion layer of some sort would work better there?

@jtopjian

@jtopjian

I feel like moving away from that is, in a way, giving up from fully understanding the implementation and ensuring API interaction is done well.

True, and you make a very good point that one of the values of Gophercloud is in fact that it handles the "hard parts".

And one more note, we do have https://github.com/gophercloud/utils which is a much more flexible code repository.

Now that's an interesting idea as sort of an "incubator" of sorts almost? Honestly most microversions for me with Cinder thus far have mostly been just "convenience" with the exception of some of the newer API's like attachments. Those are still relatively easy though because there aren't excessive versions of that, it's just it's introduction and some bug fixes along the way.

My thoughts on this today were that if there's something I REALLY need and it's not in Gophercloud I can extend my own request type in my consumer code as well as customize a response if needed (I know it's kinda lame), but the other thing is that maybe we use something like utils as a prelim for this sort of thing? Like a map of request/response types based on MV's?

As far as things "in" Gophercloud going forward I think the hardest part is the "what", implementing every mv isn't reasonable in my opinion; maybe a max version for each OpenStack Release? Then if folks need an intermediate version they can use another method to customize?

Anyway, bottom line from my perspective is that I DO think that much of the value of Gophercloud is in it's "strict" nature. My solutions thus far are just "hacks" for the very few things that I really need.

My thoughts on this today were that if there's something I REALLY need and it's not in Gophercloud I can extend my own request type in my consumer code as well as customize a response if needed (I know it's kinda lame)

It's not lame at all. The ability to do this is a core feature of Gophercloud.

As a maintainer of Gophercloud, monitoring this kind of use is important. If enough devs are doing the same extensions outside of Gophercloud, it's an indication that Gophercloud isn't meeting a core need. I think microversions fit into this area.

but the other thing is that maybe we use something like utils as a prelim for this sort of thing? Like a map of request/response types based on MV's?

I think what you're talking about here is slightly different than the idea about the "dumb" client, but this kind of client would probably also meet this need.

I think providing support for a "dumb" client is absolutely valid. I believe the Python SDK(s) provide this type of feature, so at a minimum it's valid if only to reach feature parity. An immediate benefit would be to help devs move forward with certain requests and APIs currently difficult to support in Gophercloud (ie: microversions).

I think gophercloud/utils might be the best place to house this client, though maybe it would end up going in core.

Although the client sounds simple in theory, I'd like for it to take advantage of other parts of Gophercloud core like paging and such (if possible).

As far as things "in" Gophercloud going forward I think the hardest part is the "what", implementing every mv isn't reasonable in my opinion; maybe a max version for each OpenStack Release? Then if folks need an intermediate version they can use another method to customize?

Agreed and I think it was generally agreed upon somewhere in this thread that we would base things on an OpenStack release.

But with that said, the idea I outlined above would provide the ability to support fields for every microversion. My hope is that the only barrier to entry for support is someone submitting a PR for the field to be added to the struct -- no extra methods, functions, or package extensions.

(edit: of course there might be iterations within a release that make changes to the same field a few times, possibly adding and later removing it. In this situation, basing field additions/deletions on a release is the sane thing to do)

As far as immediate next steps for me, I have some object storage work I need to finish up. After that I will work on both microversion support and the "dumb" client feature.

Finally, I will be in Vancouver, so if anyone would like to discuss this in person, let me know :)

@jtopjian

My hope is that the only barrier to entry for support is someone submitting a PR for the field to be added to the struct -- no extra methods, functions, or package extensions.

Let me try to sum it up. Structs will look like:

type Volume struct {
  Foo string `json:"foo" mv_min:"3.1" mv_max:"3.3"`
  Bar string `json:"bar" mv_min:"3.5"`
  Baz string
}

Functions will automatically compare the microversion used in client with mv_min and mv_max and behave accordingly. This will be the main approach used in approx. 95% of cases. There might be corner cases when a specific OpenStack microversion introduces specific behaviour. In such case a function will have to be modified for such corner case.

@jtopjian is that correct?

@pospispa

@pospispa Correct -- at least that's what I'm hoping 😄

We've agreed upon and merged the following document:

https://github.com/gophercloud/gophercloud/blob/master/docs/MICROVERSIONS.md

The guidelines in this document are not written in stone and might change over time. But they're a good foundation to start with.

I'm going to close this issue. I would like to thank everyone for the input they've given here.

Acc Tests: Updating LBaaS v2 Member test (gophercloud#441)

Refresh networks in host info post reconciliation (gophercloud#441)

Navigation Menu

Repository navigation

On Microversions #441

62 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

On Microversions #441

Description

Activity

jrperritt commented on Jul 31, 2017

jtopjian commented on Jul 31, 2017

pospispa commented on Aug 1, 2017

jtopjian commented on Aug 1, 2017

pospispa commented on Aug 2, 2017

pospispa commented on Aug 2, 2017

pospispa commented on Aug 2, 2017

jtopjian commented on Aug 2, 2017

pospispa commented on Aug 4, 2017

pospispa commented on Aug 4, 2017

pospispa commented on Aug 4, 2017

jtopjian commented on Aug 6, 2017

pospispa commented on Aug 7, 2017

pospispa commented on Aug 8, 2017

pospispa commented on Aug 11, 2017

62 remaining items

jtopjian commented on Nov 8, 2017

flashvoid commented on May 10, 2018

jtopjian commented on May 10, 2018

j-griffith commented on May 10, 2018

j-griffith commented on May 10, 2018

flashvoid commented on May 11, 2018

jtopjian commented on May 11, 2018

j-griffith commented on May 11, 2018

jtopjian commented on May 12, 2018

pospispa commented on May 12, 2018

jtopjian commented on May 12, 2018

jtopjian commented on Feb 6, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions