Repository navigation
Pagination is broken from messaging service (Zaqar) #3336
Description
Activity
Several tests fail because we're getting a 404 error:
message_test.go:57: Failure in message_test.go, line 57: unexpected error "Expected HTTP response code [200 204 300] when accessing [GET http://10.1.0.118/v2/queues/ACPTTESTWromD/messages?include_delayed=true&marker=3], but got 404 instead: <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL was not found on this server.</p>\n<hr>\n<address>Apache/2.4.52 (Ubuntu) Server at 10.1.0.118 Port 80</address>\n</body></html>" messaging.go:44: Attempting to delete Queue: ACPTTESTWromD messaging.go:50: Deleted Queue: ACPTTESTWromD --- FAIL: TestListMessages (0.24s)I can't exactly figure what's going on. I wonder if this has to do with openstack/zaqar@c9389d8, which changed the devstack setup and merged at the same time our failure started happening.
This line concerns me:
message_test.go:57: Failure in message_test.go, line 57: unexpected error "Expected HTTP response code [200 204 300] when accessing [GET http://10.1.0.118/v2/queues/ACPTTESTWromD/messages?include_delayed=true&marker=3], but got 404 instead: <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL was not found on this server.</p>\n<hr>\n<address>Apache/2.4.52 (Ubuntu) Server at 10.1.0.118 Port 80</address>\n</body></html>"Specifically, the URL looks off:
GET http://10.1.0.118/v2/queues/ACPTTESTWromD/messages?include_delayed=true&marker=3Based on the change you linked - specifically this line - I'd have expected this to read e.g.
GET http://10.1.0.118/messages/v2/queues/ACPTTESTWromD/messages?include_delayed=true&marker=3I wonder if our parsing of the service catalog is off...
Yeah, that looks like our issue. I deployed DevStack with the Zaqar plugin enabled and took a look at the service catalog. Zaqar is the only one that is giving a path that (I suspect?) is intended to be parsed as a relatively path or is flat out wrong:
$ curl -w "\n" http://10.0.110.161/messaging {"versions": [{"id": "1.1", "status": "DEPRECATED", "updated": "2016-7-29T02:22:47Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.messaging-v1_1+json"}], "links": [{"href": "/v1.1/", "rel": "self"}]}, {"id": "2", "status": "CURRENT", "updated": "2014-9-24T04:06:47Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.messaging-v2+json"}], "links": [{"href": "/v2/", "rel": "self"}]}]}By comparison, everything else uses fully-qualified absolute paths. For example, keystonm:
$ curl -w "\n" http://10.0.110.161/identity {"versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://10.0.110.161/identity/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}
For what it's worth, this also breaks
openstacksdkso I have no idea how their CI is currently working 😅:>>> import openstack >>> conn = openstack.connect('devstack-admin') >>> >>> conn.messaging.get('/').text '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL was not found on this server.</p>\n<hr>\n<address>Apache/2.4.58 (Ubuntu) Server at 10.0.110.161 Port 80</address>\n</body></html>\n' >>> >>> conn.identity.get('/').text '{"version": {"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://10.0.110.161/identity/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}}'
Full logs are below but IMO the fix is in Zaqar's DevStack plugin.
$ openstack catalog list +-----------------+---------------------+---------------------------------------------------------------------------+ | Name | Type | Endpoints | +-----------------+---------------------+---------------------------------------------------------------------------+ | nova | compute | RegionOne | | | | public: http://10.0.110.161/compute/v2.1 | | | | | | nova_legacy | compute_legacy | RegionOne | | | | public: http://10.0.110.161/compute/v2/3086ab003f784a49a8cacc2d4c281193 | | | | | | zaqar-websocket | messaging-websocket | RegionOne | | | | public: http://10.0.110.161:9000 | | | | | | neutron | network | RegionOne | | | | public: http://10.0.110.161:9696/networking | | | | | | glance | image | RegionOne | | | | public: http://10.0.110.161/image | | | | | | placement | placement | RegionOne | | | | public: http://10.0.110.161/placement | | | | | | zaqar | messaging | RegionOne | | | | public: http://10.0.110.161/messaging | | | | | | cinder | block-storage | RegionOne | | | | public: http://10.0.110.161/volume/v3 | | | | | | keystone | identity | RegionOne | | | | public: http://10.0.110.161/identity | | | | | +-----------------+---------------------+---------------------------------------------------------------------------+ $ curl -w "\n" http://10.0.110.161/compute/v2 {"version": {"id": "v2.0", "status": "SUPPORTED", "version": "", "min_version": "", "updated": "2011-01-21T11:33:21Z", "links": [{"rel": "self", "href": "http://10.0.110.161/compute/v2/"}, {"rel": "describedby", "type": "text/html", "href": "http://docs.openstack.org/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2"}]}} $ curl -w "\n" http://10.0.110.161:9696/networking <html> <head> <title>302 Found</title> </head> <body> <h1>302 Found</h1> The resource was found at <a href="http://10.0.110.161:9696/networking/">http://10.0.110.161:9696/networking/</a>; you should be redirected automatically. </body> </html> $ curl -w "\n" http://10.0.110.161:9696/networking/ {"versions": [{"id": "v2.0", "status": "CURRENT", "links": [{"rel": "self", "href": "http://10.0.110.161:9696/networking/v2.0/"}]}]} $ curl -w "\n" http://10.0.110.161/image {"versions": [{"id": "v2.16", "status": "SUPPORTED", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}, {"id": "v2.15", "status": "SUPPORTED", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}, {"id": "v2.14", "status": "SUPPORTED", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}, {"id": "v2.9", "status": "SUPPORTED", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}, {"id": "v2.17", "status": "CURRENT", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}, {"id": "v2.7", "status": "SUPPORTED", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}, {"id": "v2.6", "status": "SUPPORTED", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}, {"id": "v2.5", "status": "SUPPORTED", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}, {"id": "v2.4", "status": "SUPPORTED", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}, {"id": "v2.3", "status": "SUPPORTED", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}, {"id": "v2.2", "status": "SUPPORTED", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}, {"id": "v2.1", "status": "SUPPORTED", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}, {"id": "v2.0", "status": "SUPPORTED", "links": [{"rel": "self", "href": "http://10.0.110.161/image/v2/"}]}]} $ curl -w "\n" http://10.0.110.161/messaging {"versions": [{"id": "1.1", "status": "DEPRECATED", "updated": "2016-7-29T02:22:47Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.messaging-v1_1+json"}], "links": [{"href": "/v1.1/", "rel": "self"}]}, {"id": "2", "status": "CURRENT", "updated": "2014-9-24T04:06:47Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.messaging-v2+json"}], "links": [{"href": "/v2/", "rel": "self"}]}]} $ curl -w "\n" http://10.0.110.161/placement {"versions": [{"id": "v1.0", "max_version": "1.39", "min_version": "1.0", "status": "CURRENT", "links": [{"rel": "self", "href": ""}]}]} $ curl -w "\n" http://10.0.110.161/volume/v3 {"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}} $ curl -w "\n" http://10.0.110.161/volume {"versions": [{"id": "v3.0", "status": "CURRENT", "version": "3.71", "min_version": "3.0", "updated": "2023-08-31T00:00:00Z", "links": [{"rel": "describedby", "type": "text/html", "href": "https://docs.openstack.org/"}, {"rel": "self", "href": "http://10.0.110.161/volume/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=3"}]}]} $ curl -w "\n" http://10.0.110.161/identity {"versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://10.0.110.161/identity/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}
Bug report for Zaqar is here: https://bugs.launchpad.net/zaqar/+bug/2104271
Initial fix is here: https://review.opendev.org/c/openstack/zaqar/+/945589
More fixes will be needed across the API since this affects virtually all resources.
Reacted by Martin André- changed the title
[-][CI] functional-messaging job permafailing[/-][+]Pagination is broken from messaging service (Zaqar)[/+]on Jul 15, 2025 Bug report for Zaqar is here: https://bugs.launchpad.net/zaqar/+bug/2104271
Initial fix is here: https://review.opendev.org/c/openstack/zaqar/+/945589
More fixes will be needed across the API since this affects virtually all resources.
It transpires that this is not a bug in Zaqar but rather in Gophercloud. Our pagination code for the affected resources is ignoring the path of the endpoint URL, when it should be concatenating these.
https://github.com/gophercloud/gophercloud/actions/workflows/functional-messaging.yaml job is red. Fails on openstack master since Jan 1st.