Skip to content

[Bug] Leftover REV metadata not cleaned up (decomposedfs consistency errors) + backup consistency is extremely slow on large spaces, needs progress,--fast,--repair #3688

Open
Open
[Bug] Leftover REV metadata not cleaned up (decomposedfs consistency errors) + backup consistency is extremely slow on large spaces, needs progress,--fast,--repair#3688
Labels
Type:Bug

Description

@suse-coder
suse-coder
opened on Oct 9, 2026

Describe the bug

opencloud backup consistency -p /var/lib/opencloud/storage/users reports numerous inconsistencies on decomposedfs — missing symlinks, missing files, and missing blobs referenced by .REV.* node metadata. This suggests that when OpenCloud deletes a revision/blob (e.g. during version pruning or cleanup), the corresponding node metadata/reference is not cleaned up, leaving permanent dangling references that the consistency checker flags but cannot resolve on its own.

Separately, the command itself is extremely slow on larger spaces: there is no progress indication at all (no progress bar, percentage, or ETA), and every run appears to do a full rescan from scratch rather than only checking what's changed. There is also no --repair flag to automatically fix detected inconsistencies.

Steps to reproduce

  1. Have an OpenCloud deployment with larger spaces (decomposedfs backend) that have gone through normal usage (file versioning, deletions, etc.) over time.
  2. Run the consistency check against the users storage path: 
    kubectl exec -n opencloud-2 deploy/opencloud-2-opencloud -c opencloud -- opencloud backup consistency -p /var/lib/opencloud/storage/users
  3. Observe that:
    • The command takes a very long time to complete with zero progress feedback.
    • There is no way to do a faster/incremental scan (--fast) — it always seems to rescan everything.
    • There is no --repair flag to fix anything found.
  4. Review the output — it reports many [symlink missing], [files missing files missing], and [blob missing] entries, several referencing .REV.* timestamped node metadata (full output below).

Expected behavior

  • If OpenCloud deletes a revision/blob (e.g. during version pruning/cleanup), the corresponding node metadata/reference should be deleted at the same time — no dangling .REV.* references pointing at missing blobs.
  • The command should show a progress bar / percentage / processed-item counter so it's clear it's still working and how much remains, especially on large spaces.
  • A --fast flag should allow a quicker/incremental scan (e.g. skipping items unchanged since the last scan, skipping full checksum verification) instead of forcing a full rescan every time.
  • A --repair flag should allow automatically fixing detected inconsistencies where safely possible for the decomposedfs backend (e.g. recreate missing symlinks, prune orphaned blob references/metadata).

Actual behavior

  • The scan reports many inconsistent nodes and blob references, including several .REV.* timestamped blobs that are missing despite being referenced by node metadata — the metadata was not cleaned up when the blob was removed.
  • The scan takes a very long time on larger spaces with no progress indication at all.
  • Every run performs a full rescan; there's no fast/incremental mode.
  • There's no way to repair any of the found inconsistencies — manual intervention is required.

Full output for reference:

Consistency check output 
kubectl exec -n opencloud-2 deploy/opencloud-2-opencloud -c opencloud -- opencloud backup consistency -p /var/lib/opencloud/storage/users

🚨 Inconsistent Nodes:
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/9a/a4/53/18/-bd81-492f-83b6-66dc861c84ae
        👉️ [files missing files missing]        path: /var/lib/opencloud/storage/users/spaces/49/f02552-a2e5-42a3-9266-54314a566f8a/nodes/2a/71/77/68/-e186-4985-8c20-4c9c49d9e558
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/61/21/05/0c/-63da-4afa-a3b2-624e7e9f3b63
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/0e/6d/ce/17/-62e1-4715-b9c9-b1a38a40f34e
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/af/4e/fc/29/-7507-4abb-8744-aab81974c325
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/67/e0/07/24/-fb80-4938-8bae-85479e1a6304
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/65/64/29/c3/-3b3c-460c-9c16-922242f3c4da
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/43/07/12/2b/-38fc-418a-ae87-5fc808990156
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/1a/07/d1/13/-8b57-4e5f-8bb2-aece3743c1c4
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/6d/d1/0d/a5/-093a-4260-9fba-0eeba99b0f02
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/e5/da/01/cc/-e9a4-44a3-b945-df104be4eb4f
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/cb/da/d8/3b/-c3e8-4793-980f-4e6ab48c6237
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/b5/67/d2/14/-af3a-4d61-95b0-7fa11a65c6e9
        👉️ [files missing files missing]        path: /var/lib/opencloud/storage/users/spaces/49/f02552-a2e5-42a3-9266-54314a566f8a/nodes/23/b8/40/fa/-6bcb-4512-b5ad-3bbd83eca8c2
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/24/0b/67/8a/-d907-4604-8bd7-7af5723cbbcb
        👉️ [files missing files missing]        path: /var/lib/opencloud/storage/users/spaces/49/f02552-a2e5-42a3-9266-54314a566f8a/nodes/28/97/5d/9e/-f124-4615-bfb2-0620f5a720ee
        👉️ [files missing files missing]        path: /var/lib/opencloud/storage/users/spaces/49/f02552-a2e5-42a3-9266-54314a566f8a/nodes/3a/eb/ba/47/-4df9-4577-8748-d196b777027e
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/87/75/d8/f6/-c167-45f9-8a3e-2c8ce64c23a3
        👉️ [files missing files missing]        path: /var/lib/opencloud/storage/users/spaces/49/f02552-a2e5-42a3-9266-54314a566f8a/nodes/61/43/ed/12/-30b6-4db8-834c-5a0bef052d22
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/9a/06/c0/ac/-4856-4af0-86ad-401c79a15fc5
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/84/c3/b9/be/-bfef-4ff9-88b5-b365e901eebc
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/c8/0d/77/91/-e919-426b-bbb6-768f8444bdd5
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/1f/a6/e1/5a/-618a-4cb7-9573-f298650fdba7
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/28/8c/b9/b1/-2004-4609-9c1a-d90fa510d566
        👉️ [files missing files missing]        path: /var/lib/opencloud/storage/users/spaces/49/f02552-a2e5-42a3-9266-54314a566f8a/nodes/e4/a9/26/34/-2ed9-46f0-952f-dfc9e3d709bb
        👉️ [files missing files missing]        path: /var/lib/opencloud/storage/users/spaces/49/f02552-a2e5-42a3-9266-54314a566f8a/nodes/4c/b7/a2/26/-ad84-432d-aaee-d54be3f1dfbb
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/14/7a/25/82/-2b52-41e5-9bfd-cc75d02df1e9
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/5f/95/c0/d8/-3ebf-4349-8442-26a8216d4638
        👉️ [files missing files missing]        path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/62/5e/73/e7/-a8c5-437d-a07e-c7ff24841df3
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/a1/0b/52/7c/-66b2-4c18-a3b2-07c6375e18b6
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/f6/96/56/dc/-4048-4189-ac3b-e7177a163518
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/ee/05/fe/e8/-65f8-4951-a3d3-6966109c5e77
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/9b/e7/0c/2f/-fa2a-442f-aa9d-6ab3a1b63428
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/5e/88/f1/22/-3c4b-43d9-a8f8-18ecdb42ffd1
        👉️ [files missing files missing]        path: /var/lib/opencloud/storage/users/spaces/49/f02552-a2e5-42a3-9266-54314a566f8a/nodes/a0/b6/25/3d/-cf47-45b0-be97-0585e0d473b5
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/d5/9f/90/f7/-7fc9-4c04-a1ce-ca70f92b7446
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/fd/8d/42/4f/-aec9-428b-81de-151700f7fdfd
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/e6/fb/2f/35/-939a-44c5-9412-dbfe29b3d882
        👉️ [files missing files missing]        path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/78/9b/a5/02/-23e7-4233-ab4e-605dfba3346c
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/a2/fd/02/55/-d577-4692-857a-364881dd68d8
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/49/fa/d9/2c/-0767-4ea8-853b-09ca9da46fad
        👉️ [files missing files missing]        path: /var/lib/opencloud/storage/users/spaces/49/f02552-a2e5-42a3-9266-54314a566f8a/nodes/cf/64/6c/f3/-ec2a-4d3d-bfd3-04cfb24aa037
        👉️ [symlink missing]    path: /var/lib/opencloud/storage/users/spaces/73/680b56-cb7a-4676-bec7-546566f3cb61/nodes/5d/24/69/3a/-d675-405f-bfbf-a0cb79139945

🚨 Inconsistent BlobReferences:
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/5d/00/a5/fe/-2a93-40cf-805c-50b5e6c037d3
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T10:19:34.590348567Z
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/d2/d06e93-3e9f-4dd3-bf84-90780bfccdd3/blobs/3c/8c/f0/35/-21f6-4d50-a809-a22943012ee2
                                referencing node:/var/lib/opencloud/storage/users/spaces/d2/d06e93-3e9f-4dd3-bf84-90780bfccdd3/nodes/0f/a9/cc/f7/-4cc5-4068-b0cf-1ac8c5ece2df
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/5b/4d/4a/bf/-4639-451f-8ed3-499a28c336dc
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T09:47:34.555687145Z
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/2c/a3/5a/29/-2832-4ceb-8130-b3da705df53c
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T10:56:39.058794705Z.1
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/d2/d06e93-3e9f-4dd3-bf84-90780bfccdd3/blobs/fb/51/d7/74/-52d7-4def-b769-0ecd9d922fe4
                                referencing node:/var/lib/opencloud/storage/users/spaces/d2/d06e93-3e9f-4dd3-bf84-90780bfccdd3/nodes/46/e0/4b/41/-ddd9-4834-8f52-f266339026f6
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/3d/d5/b5/6e/-6b68-40b1-9b4e-6fee84c3d273
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T09:48:16.105961861Z
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/d7/b4/21/71/-54a4-499f-98fd-54dc67fcf63e
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T09:55:34.569670021Z
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/ed/ce/d1/40/-021a-413d-a58f-c0ccb2d3e182
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T09:50:58.461943662Z
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/5b/96/13/e2/-c97e-4138-b6f0-0ddb74bd0869
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T10:06:41.537282167Z
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/a8/86/6a/a3/-0051-4353-a112-bd2903cd1d41
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T09:57:34.50753013Z
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/ad/91/80/3a/-5f34-4f05-8c5f-33ed63d4c975
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T10:08:41.505412255Z
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/8f/99/b0/5c/-97ea-485b-af32-98543ecd4306
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/d2/d06e93-3e9f-4dd3-bf84-90780bfccdd3/blobs/7b/e3/5f/a6/-601c-436a-8e21-ea2709710182
                                referencing node:/var/lib/opencloud/storage/users/spaces/d2/d06e93-3e9f-4dd3-bf84-90780bfccdd3/nodes/db/1d/93/c4/-69e6-4fe6-a8e4-1a073282634e.REV.2026-09-07T12:42:17.955380662Z.1
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/3d/1a/b3/15/-1500-4f28-8a91-a084cff742dd
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T09:49:34.488868365Z
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/83/5c/43/93/-3185-4100-a754-a5fda96c45bb
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T10:11:34.583202041Z
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/e7/53/65/29/-e831-4d55-8733-ab5ec0596698
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T09:52:16.096286941Z
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/d2/d06e93-3e9f-4dd3-bf84-90780bfccdd3/blobs/c0/c8/96/0c/-d250-46b1-be74-889e12ce400b
                                referencing node:/var/lib/opencloud/storage/users/spaces/d2/d06e93-3e9f-4dd3-bf84-90780bfccdd3/nodes/c1/31/1d/fa/-67d0-4a0a-9029-e4e113e86814
        👉️ [blob missing]       blob: /var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/blobs/d5/0a/c3/71/-6b0a-4a5e-8446-a5ecf8161843
                                referencing node:/var/lib/opencloud/storage/users/spaces/a7/828a78-932f-45e5-bac9-05e113463317/nodes/7d/84/88/69/-305e-433d-9879-2e0fff23f510.REV.2026-10-09T09:57:00.498437389Z

Questions for maintainers

  • Why are these .REV.* blobs missing in the first place — is this caused by automatic revision pruning/cleanup, a retention policy, quota-triggered cleanup, or something else?
  • If OpenCloud itself deletes a blob (e.g. during revision cleanup), why is the corresponding revision metadata/reference not also removed at the same time? This leaves dangling references that the consistency checker flags but cannot resolve.
  • Is this expected under normal operation (e.g. concurrent cleanup + backup run), or is this indicative of a deeper bug in decomposedfs revision/blob lifecycle management?

Setup

Please describe how you started the server and provide a list of relevant environment variables or configuration files.

Details 

OC_XXX=somevalue
OC_YYY=somevalue
PROXY_XXX=somevalue

Additional context

  • OpenCloud version: latest (8.1.0)
  • Storage backend: decomposedfs
  • Deployed via Kubernetes
  • Command used: 
    kubectl exec -n opencloud-2 deploy/opencloud-2-opencloud -c opencloud -- opencloud backup consistency -p /var/lib/opencloud/storage/users
  • Requested improvements:
    1. Root-cause fix: ensure node metadata/references are cleaned up whenever OpenCloud deletes the corresponding revision/blob, so these inconsistencies don't occur in the first place.
    2. Progress bar / live progress output (e.g. "scanned X/Y nodes", percentage, ETA) for backup consistency.
    3. --fast flag for a lighter/quicker scan that doesn't require a full re-scan of everything every time (e.g. skip unchanged items since last run, skip full checksum verification).
    4. --repair flag to automatically fix detected inconsistencies where safely possible for the decomposedfs backend (e.g. recreate missing symlinks, prune orphaned blob references/metadata).

Activity

  1. suse-coder
    added
    Type:Bug
    on Oct 9, 2026
  2. github-project-automation
    added this to OpenCloud Team Boardon Oct 9, 2026
  3. suse-coder
    changed the title [-][Bug]` Leftover REV metadata not cleaned up (decomposedfs consistency errors) + `backup consistency` is extremely slow on large spaces, needs progress/`--fast`/`--repair[/-] [+][Bug] Leftover REV metadata not cleaned up (decomposedfs consistency errors) + backup consistency is extremely slow on large spaces, needs progress,--fast,--repair[/+] on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type:Bug

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions