Error when querying images: "json: cannot unmarshal string into Go struct field .images of type bool"

Description

I'm encountering an error when querying images using the terraform-provider-openstack v2.1.0 (older ones didnt work as well) with Gophercloud v1.13.0. The error message indicates that there is an issue with unmarshalling JSON response into the Go struct, specifically json: cannot unmarshal string into Go struct field .images of type bool.

Error Log

Here is the relevant part of the error log:

X-Openstack-Request-Id: req-810f3056-f3b9-47e1-90d7-d56ed14fb69a: timestamp="2024-07-23T14:47:56.745+0200"
2024-07-23T14:47:56.746+0200 [INFO]  provider.terraform-provider-openstack_v2.1.0: 2024/07/23 14:47:56 [DEBUG] OpenStack Response Body: {
  "first": "/v2/images?sort=name%3Aasc\u0026status=active\u0026name=Ubuntu+24.04+LTS",
  "images": [
    {
      "architecture": "x86_64",
Cache-Control: no-cache
User-Agent: HashiCorp Terraform/1.9.2 (+https://www.terraform.io) Terraform Plugin SDK/v2.30.0 Terraform Provider OpenStack/2.1.0 gophercloud/v1.13.0
X-Auth-Token: ***: timestamp="2024-07-23T14:47:56.545+0200"
2024-07-23T14:47:56.745+0200 [INFO]  provider.terraform-provider-openstack_v2.1.0: 2024/07/23 14:47:56 [DEBUG] OpenStack Response Code: 200: timestamp="2024-07-23T14:47:56.745+0200"
2024-07-23T14:47:56.745+0200 [INFO]  provider.terraform-provider-openstack_v2.1.0: 2024/07/23 14:47:56 [DEBUG] OpenStack Response Headers:
Connection: keep-alive
Content-Length: 1636
Content-Type: application/json; charset=UTF-8
Date: Tue, 23 Jul 2024 12:47:56 GMT
Front-End-Https: on
Server: nginx
X-Openstack-Request-Id: req-810f3056-f3b9-47e1-90d7-d56ed14fb69a: timestamp="2024-07-23T14:47:56.745+0200"
2024-07-23T14:47:56.746+0200 [INFO]  provider.terraform-provider-openstack_v2.1.0: 2024/07/23 14:47:56 [DEBUG] OpenStack Response Body: {
  "first": "/v2/images?sort=name%3Aasc\u0026status=active\u0026name=Ubuntu+24.04+LTS",
  "images": [
    {
      "architecture": "x86_64",
      "checksum": "6785871c573e1ac9ced4c6e3ba8026d4",
      "container_format": "bare",
      "created_at": "2024-06-25T11:39:05Z",
      "direct_url": "rbd://3ed57603-5970-4c06-8fb2-d383540a88a1/glance-images/4e0b34b4-ee30-4384-a821-ef41828d3143/snap",
      "disk_format": "raw",
      "file": "/v2/images/4e0b34b4-ee30-4384-a821-ef41828d3143/file",
      "filesystems": "ext4",
      "hw_disk_bus": "scsi",
      "hw_qemu_guest_agent": "true",
      "hw_scsi_model": "virtio-scsi",
      "id": "4e0b34b4-ee30-4384-a821-ef41828d3143",
      "locations": [
        {
          "metadata": {},
          "url": "rbd://3ed57603-5970-4c06-8fb2-d383540a88a1/glance-images/4e0b34b4-ee30-4384-a821-ef41828d3143/snap"
        },
        {
          "metadata": {},
          "url": "rbd://65f55c31-8c1a-4294-938f-51c260f84c21/glance-images/4e0b34b4-ee30-4384-a821-ef41828d3143/snap"
        },
        {
          "metadata": {},
          "url": "rbd://05b72e9b-fd8c-4d62-8cd3-8ec6b7ef6627/glance-images/4e0b34b4-ee30-4384-a821-ef41828d3143/snap"
        }
      ],
      "min_disk": 10,
      "min_ram": 512,
      "name": "Ubuntu 24.04 LTS",
      "orig_id": "4e0b34b4-ee30-4384-a821-ef41828d3143",
      "os_distro": "ubuntu",
      "os_hidden": "False",
      "os_type": "linux",
      "os_visibility": "public",
      "owner": "de8967bdfa1c4e06ac905a1331f1e71e",
      "owner_specified.openstack.object": "images/Ubuntu 24.04 LTS (20240625113902)",
      "project": "de8967bdfa1c4e06ac905a1331f1e71e",
      "protected": false,
      "schema": "/v2/schemas/image",
      "self": "/v2/images/4e0b34b4-ee30-4384-a821-ef41828d3143",
      "size": 10737418240,
      "status": "active",
      "tags": [],
      "updated_at": "2024-07-23T11:50:10Z",
      "virtual_size": null,
      "visibility": "public"
    }
  ],
  "schema": "/v2/schemas/images"
}: timestamp="2024-07-23T14:47:56.745+0200"


(...)

2024-07-23T14:47:56.746+0200 [ERROR] provider.terraform-provider-openstack_v2.1.0: Response contains error diagnostic: diagnostic_detail="" diagnostic_severity=ERROR tf_data_source_type=openstack_images_image_v2 tf_provider
_addr=registry.terraform.io/terraform-provider-openstack/openstack @module=sdk.proto tf_proto_version=5.4 tf_req_id=282f70e3-6ebe-218b-2137-deec11aba05d tf_rpc=ReadDataSource @caller=github.com/hashicorp/terraform-plugin-go
@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_summary="Unable to query images: json: cannot unmarshal string into Go struct field .images of type bool" timestamp="2024-07-23T14:47:56.746+0200"
2024-07-23T14:47:56.747+0200 [ERROR] vertex "data.openstack_images_image_v2.ubuntu" error: Unable to query images: json: cannot unmarshal string into Go struct field .images of type bool
2024-07-23T14:47:56.747+0200 [ERROR] vertex "data.openstack_images_image_v2.ubuntu (expand)" error: Unable to query images: json: cannot unmarshal string into Go struct field .images of type bool

Failing Image Data

Here is the data for the failing image query:

{
  "checksum": "6785871c573e1ac9ced4c6e3ba8026d4",
  "container_format": "bare",
  "created_at": "2024-06-25T11:39:05Z",
  "disk_format": "raw",
  "file": "/v2/images/4e0b34b4-ee30-4384-a821-ef41828d3143/file",
  "id": "4e0b34b4-ee30-4384-a821-ef41828d3143",
  "min_disk": 10,
  "min_ram": 512,
  "name": "Ubuntu 24.04 LTS",
  "owner": "1870421f338248f684e2b1c0cc63b5d4",
  "properties": {
    "os_hidden": true,
    "os_visibility": "public",
    "owner_specified.openstack.sha256": "",
    "orig_id": "4e0b34b4-ee30-4384-a821-ef41828d3143",
    "owner_specified.openstack.object": "images/Ubuntu 24.04 LTS (20240625113902)",
    "filesystems": "ext4",
    "project": "de8967bdfa1c4e06ac905a1331f1e71e",
    "owner_specified.openstack.md5": "",
    "locations": [
      {
        "url": "rbd://3ed57603-5970-4c06-8fb2-d383540a88a1/glance-images/4e0b34b4-ee30-4384-a821-ef41828d3143/snap",
        "metadata": {}
      },
      {
        "url": "rbd://65f55c31-8c1a-4294-938f-51c260f84c21/glance-images/4e0b34b4-ee30-4384-a821-ef41828d3143/snap",
        "metadata": {}
      },
      {
        "url": "rbd://05b72e9b-fd8c-4d62-8cd3-8ec6b7ef6627/glance-images/4e0b34b4-ee30-4384-a821-ef41828d3143/snap",
        "metadata": {}
      }
    ],
    "direct_url": "rbd://3ed57603-5970-4c06-8fb2-d383540a88a1/glance-images/4e0b34b4-ee30-4384-a821-ef41828d3143/snap",
    "architecture": "x86_64",
    "os_distro": "ubuntu",
    "hw_disk_bus": "scsi",
    "hw_scsi_model": "virtio-scsi",
    "os_type": "linux",
    "hw_qemu_guest_agent": "yes"
  },
  "protected": false,
  "schema": "/v2/schemas/image",
  "size": 10737418240,
  "status": "active",
  "tags": [],
  "updated_at": "2024-07-22T15:07:50Z",
  "visibility": "public"
}

Working Image Data

Here is the data for a working image query for comparison:

{
  "container_format": "bare",
  "created_at": "2023-01-04T10:49:23Z",
  "disk_format": "raw",
  "file": "/v2/images/6e4b4289-22d2-4c8b-8a61-18266e4f1a04/file",
  "id": "6e4b4289-22d2-4c8b-8a61-18266e4f1a04",
  "min_disk": 8,
  "min_ram": 0,
  "name": "Ubuntu 22.04 (LTS)",
  "owner": "de8967bdfa1c4e06ac905a1331f1e71e",
  "properties": {
    "image_supports_password": "true",
    "vendor_is_eol_after": "2027/04/02",
    "description": "Ubuntu 22.04 (Jammy Jellyfish)",
    "filesystems": "ext4",
    "image_supports_keypair": "true",
    "locations": [
      {
        "url": "rbd://05b72e9b-fd8c-4d62-8cd3-8ec6b7ef6627/glance-images/6e4b4289-22d2-4c8b-8a61-18266e4f1a04/snap",
        "metadata": {}
      },
      {
        "url": "rbd://65f55c31-8c1a-4294-938f-51c260f84c21/glance-images/6e4b4289-22d2-4c8b-8a61-18266e4f1a04/snap",
        "metadata": {}
      },
      {
        "url": "rbd://3ed57603-5970-4c06-8fb2-d383540a88a1/glance-images/6e4b4289-22d2-4c8b-8a61-18266e4f1a04/snap",
        "metadata": {}
      }
    ],
    "direct_url": "rbd://05b72e9b-fd8c-4d62-8cd3-8ec6b7ef6627/glance-images/6e4b4289-22d2-4c8b-8a61-18266e4f1a04/snap",
    "architecture": "x86_64",
    "os_distro": "ubuntu",
    "os_version": "22.04",
    "hw_disk_bus": "scsi",
    "hw_scsi_model": "virtio-scsi",
    "os_type": "linux",
    "hw_qemu_guest_agent": "yes",
    "os_require_quiesce": true
  },
  "protected": false,
  "schema": "/v2/schemas/image",
  "size": 8589934592,
  "status": "active",
  "tags": [],
  "updated_at": "2023-08-28T08:34:59Z",
  "visibility": "public"
}

Steps to Reproduce

Use the terraform-provider-openstack v2.1.0 to query an image with the specified failing data.
Observe the error message indicating a JSON unmarshalling issue.

Expected Behavior

The query should successfully retrieve the image information without any unmarshalling errors.

Actual Behavior

The query fails with an error message stating json: cannot unmarshal string into Go struct field .images of type bool.

Thank you for reporting your first issue! Be sure that we will be looking at it, but keep in mind
this sometimes takes a while.
Please let the maintainers know if your issue has not got enough attention after a few days.
If any doubt, please consult our issue tutorial.

@kayrus

@kayrus does this ring a bell?

We have a similar issue terraform-provider-openstack/terraform-provider-openstack#1706

@kayrus

@kayrus are you far enough in the investigation to say whether Gophercloud is the cause?

@pierreprinetti

It looks like anyone can (or was able) submit a custom property to an image and its value will overlap the system value.

I tried to reproduce this locally, but I got an error from API:

Provided object does not match schema 'image': 'False' is not of type 'boolean' Failed validating 'type' in schema['properties']['os_hidden']: {'description': 'If true, image will not appear in default image list ' 'response.', 'type': 'boolean'} On instance['os_hidden']: 'False'

I'm not sure whether we need to fix the unmarshaller. If we need to do this, then we need to unmarshal any field to an interface.

I drafted a fix for this particular issue:

diff --git a/openstack/image/v2/images/results.go b/openstack/image/v2/images/results.go
index 6652f0e7..93bec50c 100644
--- a/openstack/image/v2/images/results.go
+++ b/openstack/image/v2/images/results.go
@@ -4,6 +4,7 @@ import (
        "encoding/json"
        "fmt"
        "reflect"
+       "strconv"
        "strings"
        "time"
 
@@ -102,6 +103,7 @@ func (r *Image) UnmarshalJSON(b []byte) error {
        type tmp Image
        var s struct {
                tmp
+               Hidden                      any    `json:"os_hidden"`
                SizeBytes                   any    `json:"size"`
                OpenStackImageImportMethods string `json:"openstack-image-import-methods"`
                OpenStackImageStoreIDs      string `json:"openstack-image-store-ids"`
@@ -123,6 +125,20 @@ func (r *Image) UnmarshalJSON(b []byte) error {
                return fmt.Errorf("Unknown type for SizeBytes: %v (value: %v)", reflect.TypeOf(t), t)
        }
 
+       switch t := s.Hidden.(type) {
+       case nil:
+               r.Hidden = false
+       case bool:
+               r.Hidden = t
+       case string:
+               r.Hidden, err = strconv.ParseBool(t)
+               if err != nil {
+                       return fmt.Errorf("Failed to parse Hidden %q: %v", t, err)
+               }
+       default:
+               return fmt.Errorf("Unknown type for Hidden: %v (value: %v)", reflect.TypeOf(t), t)
+       }
+
        // Bundle all other fields into Properties
        var result any
        err = json.Unmarshal(b, &result)
@@ -131,6 +147,7 @@ func (r *Image) UnmarshalJSON(b []byte) error {
        }
        if resultMap, ok := result.(map[string]any); ok {
                delete(resultMap, "self")
+               delete(resultMap, "os_hidden")
                delete(resultMap, "size")
                delete(resultMap, "openstack-image-import-methods")
                delete(resultMap, "openstack-image-store-ids")

As for the properties in terraform-provider-openstack/terraform-provider-openstack#1706, we may convert them from map[string]string to map[string]any. @pierreprinetti thoughts?

UPD: Properties are already in map[string]any format. But looks like we need to add an extra logic into the unmarshaller.

its value will overlap the system value

My 2¢:

If you want to use this EVIL feature of Glance to override existing fields, it's on you and you can use your custom Image struct to marshal/unmarshal with ExtractInto & co.

Am I being too harsh? Are there valid use-cases for this sorcery?

@pierreprinetti

@pierreprinetti let's just cover these two cases, and if there will be another occurrence, then we'll just give an advice to use a custom struct. Unfortunately with the terraform provider we need to add a new field every time we receive a bug report. Unless we invent a smart way for handling this automatically.

@stephenfin

@stephenfin I recall you recently dealt with this problem in Gophercloud of metadata mixing up with regular fields. Do you have any insight to share?

I assume you're talking about the server scheduler hints rework? If so, that's not really related: in that, I was merely taking a second top level key and exposing it as a second variable to the server create call, rather than munging it in.

Unfortunately the way we solve this in openstacksdk is seemingly the same as it is here, namely to pull out well known properties to top level keys but leave everything else inside a properties dictionary which is free-form. Because it's Python, types are less of an issue also 😄. So I've no good ideas for you, I'm afraid

Navigation Menu

Repository navigation

Error when querying images: "json: cannot unmarshal string into Go struct field .images of type bool" #3147

Description

Error Log

Failing Image Data

Working Image Data

Steps to Reproduce

Expected Behavior

Actual Behavior

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Error when querying images: "json: cannot unmarshal string into Go struct field .images of type bool" #3147

Description

Description

Error Log

Failing Image Data

Working Image Data

Steps to Reproduce

Expected Behavior

Actual Behavior

Activity

github-actions commented on Jul 23, 2024

pierreprinetti commented on Jul 26, 2024

kayrus commented on Jul 26, 2024

pierreprinetti commented on Jul 26, 2024

kayrus commented on Jul 26, 2024

pierreprinetti commented on Jul 26, 2024

kayrus commented on Jul 26, 2024

pierreprinetti commented on Jul 26, 2024

stephenfin commented on Jul 27, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions