feat: Add version check mechanism to gRPC calls by jabr · Pull Request #260 · psviderski/uncloud

jabr · 2026-03-04T07:23:06Z

This adds a minimum version mechanism to gRPC communication to ensure CLI and daemon versions are compatible. It enables future breaking changes by allowing minimum version requirements to be enforced.

See #247 for example use-case.

Changes:

New pkg/versioncheck package with gRPC interceptors
Client interceptors inject version metadata (x-uncloud-cli-version, x-uncloud-min-daemon-version) into requests
Server interceptors validate client version headers and respond with x-uncloud-daemon-version
Initial minimum versions set to "0.0.0" for backward compatibility with pre-version-check clients/daemons
After first real min-version bump occurs, client warns if daemon response is missing version metadata (useful during initial version-check release transition; once user has upgraded to version checking daemons it will preemptively reject the client request)
- In the current version, at min 0.0.0, it doesn't warn in order to not be unnecessarily annoying since they are presumably compatible versions. The hope is that people would have upgraded both client and daemon sometime before the first real min-version bump is necessary. But we could change this to start warning right away when a new client is used with a old server in order to prompt people to upgrade...

Behavior:

Server rejects requests if CLI version is below minimum or daemon version is below CLI's required minimum
Missing version headers are treated as "0.0.0", which passes initial checks but will fail once minimums are bumped in future releases

Files changed:

pkg/versioncheck/interceptor.go - new package with interceptors
pkg/client/connector/*.go - added client interceptors
internal/machine/machine.go - added server interceptors to proxy servers

jabr · 2026-03-06T05:30:44Z

I've tested this on my cluster against a variety of combinations: old/current (no version check logic) client and server with new (with version check logic) client and server, as well as different future cases with different version dependency and current version combinations in client and server.

Note that this initial release has min client/daemon version requirements of 0.0.0 and it treats old clients/daemons with no version specified as implicitly 0.0.0 to make it backwards compatible for the initial release of this.

old client <-> old server [ok, obviously]

$ uc ls
NAME          MODE         REPLICAS   IMAGE              ENDPOINTS
caddy         global       1          caddy:latest       (custom Caddy config)

min-versions 0.0.0 [all ok for backwards compatibility on initial release]

new client <-> old server [ok]

$ ./dist/uncloud ls
NAME          MODE         REPLICAS   IMAGE              ENDPOINTS
caddy         global       1          caddy:latest       (custom Caddy config)

old client <-> new server [ok]

$ uc ls
NAME          MODE         REPLICAS   IMAGE              ENDPOINTS
caddy         global       1          caddy:latest       (custom Caddy config)

new client <-> new server [ok]

$ ./dist/uncloud ls
NAME          MODE         REPLICAS   IMAGE              ENDPOINTS
caddy         global       1          caddy:latest       (custom Caddy config)

min-versions 0.0.1 (or whatever the first real compatibility version break will be)

new client <-> old server [warning because version was unchecked, but no way to prevent what happened]

$ ./dist/uncloud ls
WARNING: daemon version is below minimum required version 0.0.1. The daemon did not verify this CLI's minimum version requirement, so the operation may not have behaved as intended. Please upgrade the daemon: https://github.com/psviderski/uncloud/releases/latest
NAME          MODE         REPLICAS   IMAGE              ENDPOINTS
caddy         global       1          caddy:latest       (custom Caddy config)

old client <-> new server [incompatible]

$ uc ls
Error: list services: list machines: rpc error: code = FailedPrecondition desc = version check failed: client version is below minimum 0.0.1. Please upgrade: https://github.com/psviderski/uncloud/releases/latest

new client <-> new server [ok]

$ ./dist/uncloud ls
NAME          MODE         REPLICAS   IMAGE              ENDPOINTS
caddy         global       1          caddy:latest       (custom Caddy config)

min-client-version > client version [incompatible according to daemon]

$ ./dist/uncloud ls
Error: list services: list machines: rpc error: code = FailedPrecondition desc = version check failed: client version is below minimum 999.0.0. Please upgrade: https://github.com/psviderski/uncloud/releases/latest

min-daemon-version > daemon version [incompatible according to client]

$ ./dist/uncloud ls
Error: list services: list machines: rpc error: code = FailedPrecondition desc = version check failed: daemon version 999.0.0-dev is below client's minimum required version 999.0.0. Please upgrade the daemon: https://github.com/psviderski/uncloud/releases/latest

jabr · 2026-03-06T06:07:18Z

Also, tested the streaming case:

new client, min-daemon=0.0.0 <-> old server

$ ./dist/uncloud logs wernicke -f
Mar  5 21:09:49.836 himalia wernicke[2e370] INFO:wernicke:→ GET /health | 200 (15)
Mar  5 21:10:19.905 himalia wernicke[2e370] INFO:wernicke:→ GET /health | 200 (15)
(15)

new client, min-daemon=0.0.1 <-> old server

$ ./dist/uncloud logs wernicke -f
WARNING: daemon version is below minimum required version 0.0.1. The daemon did not verify this CLI's minimum version requirement, so the operation may not have behaved as intended. Please upgrade the daemon: https://github.com/psviderski/uncloud/releases/latest
Mar  5 21:07:49.572 himalia wernicke[2e370] INFO:wernicke:→ GET /health | 200 (15)
Mar  5 21:08:19.635 himalia wernicke[2e370] INFO:wernicke:→ GET /health | 200 (15)

old client <-> new server, min-client=0.0.1

$ uc logs wernicke -f
Error: stream logs for service 'wernicke': inspect service: list machines: rpc error: code = FailedPrecondition desc = version check failed: client version is below minimum 0.0.1. Please upgrade: https://github.com/psviderski/uncloud/releases/latest

new client <-> new server, min-client=999.0.0

$ ./dist/uncloud logs wernicke -f
Error: stream logs for service 'wernicke': inspect service: list machines: rpc error: code = FailedPrecondition desc = version check failed: client version is below minimum 999.0.0. Please upgrade: https://github.com/psviderski/uncloud/releases/latest

new client, min-daemon=999.0.0 <-> new server

$ ./dist/uncloud logs wernicke -f
Error: stream logs for service 'wernicke': inspect service: list machines: rpc error: code = FailedPrecondition desc = version check failed: daemon version 999.0.0-dev is below client's minimum required version 999.0.0. Please upgrade the daemon: https://github.com/psviderski/uncloud/releases/latest

psviderski

I'm sorry it took that long for me to take a look at this PR.

Overall it's clear what it does but I'm not sure I understand the idea behind MinCLIVersion and MinDaemonVersion.
Why would you want to set them globally for a client or server/daemon?

I wonder if we can make it simpler and stateless.

I can think of only one use case for this: when evolving the API and adding a new method, we want the client to not fail with Unimplemented. But instead fail with a human-friendly FailedPrecondition error that says that version X.Y.Z is required on the server, please upgrade.

Let's say the current version is 0.17.1, we're adding a new method Foo that is going to be released in 0.18.0. At the beginning of the Foo method in the client we will add something like this:

ctx = metadata.AppendToOutgoingContext(ctx, MetadataKeyMinDaemonVersion, "0.18.0")

When the user then use an updated uc CLI to 0.18.0 against a machine that is still on <0.18.0, the commands that use the new method Foo will fail with an error that minimal daemon version is not satisfied for this command. But other older commands will continue to work.

Some CLI commands may also check the server version and branch the logic based on it if needed, e.g. use old deprecated methods for older servers but new methods for new ones.

Ah, I just recalled the big change to the proxy layer to use a new format for machines header. I guess this is the case when you'd consider setting global min version, right?

I wonder whether this is still needed. There are two cases:

old client -> new server
On the server we can assume new clients use the new machines format, so the director can check if the client is >=0.18.0 then parse the new format. If missing, then use the old ipv6 format. So still support both for a while.
new client -> old server
We can refactor the code so that we use only ProxyMachinesContext everywhere (replace proxyToMachine). Then make some direct grpc call from ProxyMachinesContext to extract the version of the server from the header in the response. Then based on it decide what machines format to use. So basically still have the current logic of resolving names to IPs as a fallback for the old servers.

This looks like we can even make the change to the proxy backward compatible this way if I'm not missing anything. But as I mentioned earlier it might not worth it.

I was just thinking out loud. I hope this makes sense. Please let me know what you think when you have time.

psviderski · 2026-03-25T07:29:49Z

 			proxyServer := grpc.NewServer(
 				grpc.ForceServerCodecV2(proxy.Codec()),
+				grpc.UnaryInterceptor(versionpkg.ServerUnaryInterceptor),
+				grpc.StreamInterceptor(versionpkg.ServerStreamInterceptor),


Not sure I understand why we put server interceptors on the proxies. Especially this non-local one that can proxy to different gRPC servers which could have different versions.

I'd put them on the local machine server:

uncloud/internal/machine/machine.go

Line 303 in 51c7b5b

m.localMachineServer = newGRPCServer(m, c, m.dockerServer, caddyServer)

The actual server that responds, not the proxies that just relay messages so that no matter where the request comes from, it goes through the interceptors on each machine. Am I missing something?

This isn't necessary for the proxy refactor change, but it could be relevant for other kinds of backwards-compatibility breaking changes.

For example, if the semantics of a gRPC function itself changed, then it's the target daemon we are proxying to that needs to be checking versions as well.

The client has a min-daemon of "1.2" and connects to a machine running daemon "1.2" (so that check passes) but it wants the command proxy'd to another machine. And that machine is running daemon "1.1" still, so it needs to check the version, too.

Otherwise the new "foobar" implementation in 1.2 that the client is expecting is going to actually get the old 1.1 response. 😄

And that machine is running daemon "1.1" still, so it needs to check the version, too.

Still not following why this should be checked at the proxy, not at the server on the machine with 1.1 version.

For example, if the semantics of a gRPC function itself changed, then it's the target daemon we are proxying to that needs to be checking versions as well.

I think I know what you're talking about but that should be a super rare case. Changing the semantics of an API call is shooting yourself in the foot and asking for trouble. A changed semantics should usually be implemented as a new method to not deal with any versioning, only the minimal version the method was added in.

Still not following why this should be checked at the proxy, not at the server on the machine with 1.1 version.

Ah, I see. It's my understanding that the client connection (or remote server proxying to it) comes in at this proxy server (and then routes it to its local machine server and/or remote servers' proxy server entry point).

So we need to do the version check there -- at the proxy entry point and before we get to the local machine server -- in order to make use of it for changes to the proxy routing logic itself (like we need for the proxy refactor branch).

It's entirely possible I missed something here, though, and have the connection flow wrong.

Also, we might need to add it to the local machine server, as well, if it is possible something connects directly to it and not via its proxy server.

jabr · 2026-03-25T15:24:25Z

Ah, I just recalled the big change to the proxy layer to use a new format for machines header. I guess this is the case when you'd consider setting global min version, right?

Yes, this was intended for complex, backwards incompatible changes, possibly affecting many commands. A change like the proxy refactor doesn't just throw errors when things are mismatched, it can silently do the wrong thing in some cases. For example, it could run a command on a different server than the user intended and return that result with no obvious errors or exceptions.

It was also meant to be a general solution for changes like it that may be required in the future. It's certainly overkill for something as simple as a new gRPC method missing, but it would enable forcing an upgrade if we ever need to change the "semantics" of an existing call. e.g. something with the deploy process that would be similarly painful to support multiple versions of for the sake of backwards compatibility.

Also, this implementation has some bits that are dealing with the initial transition from no version info available, which is why things like the client check on the server response is needed to at least warn the user after the fact.

Then make some direct grpc call from ProxyMachinesContext to extract the version of the server from the header in the response.

That's the pre-flighting approach, which I was trying to avoid needing with this implementation. For the proxy refactor, that means a preflight check on a significant fraction of all calls the the cli makes.

jabr · 2026-03-25T15:39:50Z

Also, for clarity, the reason for the two different "min-versions":

client sends min-daemon-version

This is so we don't need to preflight requests: the newer client code knows it needs a daemon of at least version x.y.z to handle its request properly. And any version-checking daemon is capable of recognizing it is below the threshold and can then abort with an immediate error.

server checks min-client-version

This is so the server can just tell the client it is too old and no longer supported. We could still support a transition phase of backwards compatibility with the server having conditional responses based on the client version (e.g. return v0.1 payload, v0.2 payload, etc).

But when we finally drop support for an old client (whether that is immediately or eventual), the min-client-version gives us a cut-off where we can safely know that old clients will just stop working until upgraded.

psviderski · 2026-03-26T05:26:40Z

I'm happy to merge this if we can remove the checks that are necessary only for the transition period after a few more releases. Adding TODOs for this and transferring explanations from here to the code comments will also be helpful, especially the version constants in interceptor.go and what each interceptor checks.

jabr · 2026-03-26T08:17:26Z

I'm happy to merge this if we can remove the checks that are necessary only for the transition period after a few more releases. Adding TODOs for this and transferring explanations from here to the code comments will also be helpful, especially the version constants in interceptor.go and what each interceptor checks.

Sounds good. I still have a few more follow-ups to make and then re-run my manual tests (hopefully in the next day or two).

…ed by both pkg/api/ and pkg/client/

…can be removed eventually after transition to version checking client/daemons

jabr · 2026-04-01T06:06:58Z

Okay, I believe have made all of the updates we discussed. Let me know if you have any other questions, particularly about the server interceptors on the proxy endpoints.

psviderski

Thank you for the updates. Will resolve conflicts and merge

jabr marked this pull request as draft March 4, 2026 07:23

jabr mentioned this pull request Mar 4, 2026

refactor: Improvements to machine proxy calls and response metadata #247

Merged

jabr marked this pull request as ready for review March 6, 2026 05:42

jabr changed the title ~~Add version check mechanism to gRPC calls~~ feat: Add version check mechanism to gRPC calls Mar 6, 2026

psviderski reviewed Mar 25, 2026

View reviewed changes

psviderski reviewed Mar 26, 2026

View reviewed changes

Comment thread pkg/versioncheck/interceptor.go

jabr added 13 commits March 31, 2026 22:11

Add version check mechanism to gRPC calls

889d295

Use semver not semver/v3

7fa6ac1

Give dev builds a special "infinite" version number (999.0.0-dev)

6564d31

Handle no metadata on grpc call correctly for version check

5690201

Move versioncheck package to root pkg/ from pkg/api/ since it is shar…

8223fd7

…ed by both pkg/api/ and pkg/client/

Only show "no daemon version" warning once

f2946b4

Append version headers to metadata, not overwrite...

0f8b8f7

Unit tests on versioncheck logic

9dcc035

Move SetHeader to the ServerStream in ServerStreamInterceptor

0ad22ae

Go modernizer nits: interface{} -> any

5ea1466

Use more conventional gRPC header names for version/min-versions

2c9c552

Add TODO notes on checkDaemonVersionInResponse and related code that …

d9f70b1

…can be removed eventually after transition to version checking client/daemons

Use testify for testing assertions

53b0e6a

jabr force-pushed the version-check branch from 391f2d5 to e9246cc Compare April 1, 2026 06:02

Add explanatory comments on MinCLIVersion and MinDaemonVersion

3594766

jabr force-pushed the version-check branch from e9246cc to 3594766 Compare April 1, 2026 06:05

jabr requested a review from psviderski April 1, 2026 06:07

psviderski approved these changes Apr 8, 2026

View reviewed changes

Merge branch 'main' into version-check

f4fa5af

psviderski merged commit 897f30f into psviderski:main Apr 8, 2026
3 of 4 checks passed

Conversation

jabr commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jabr commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

old client <-> old server [ok, obviously]

min-versions 0.0.0 [all ok for backwards compatibility on initial release]

new client <-> old server [ok]

old client <-> new server [ok]

new client <-> new server [ok]

min-versions 0.0.1 (or whatever the first real compatibility version break will be)

new client <-> old server [warning because version was unchecked, but no way to prevent what happened]

old client <-> new server [incompatible]

new client <-> new server [ok]

min-client-version > client version [incompatible according to daemon]

min-daemon-version > daemon version [incompatible according to client]

Uh oh!

jabr commented Mar 6, 2026

new client, min-daemon=0.0.0 <-> old server

new client, min-daemon=0.0.1 <-> old server

old client <-> new server, min-client=0.0.1

new client <-> new server, min-client=999.0.0

new client, min-daemon=999.0.0 <-> new server

Uh oh!

psviderski left a comment

Choose a reason for hiding this comment

Uh oh!

psviderski Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

jabr Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

psviderski Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

jabr Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jabr commented Mar 25, 2026

Uh oh!

jabr commented Mar 25, 2026

Uh oh!

Uh oh!

psviderski commented Mar 26, 2026

Uh oh!

jabr commented Mar 26, 2026

Uh oh!

jabr commented Apr 1, 2026

Uh oh!

psviderski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jabr commented Mar 4, 2026 •

edited

Loading

jabr commented Mar 6, 2026 •

edited

Loading

jabr Mar 26, 2026 •

edited

Loading