Skip to content

After deleting image, the preview files remain forever #11574

Closed
Closed
After deleting image, the preview files remain forever#11574
Labels
0. Needs triagePending check for reproducibility or if it fits our roadmapbugfeature: previews and thumbnailsneeds infostaleTicket or PR with no recent activity

Description

@ho1ger
ho1ger
opened on Oct 3, 2018

This issue is obviously related to #5558, which was closed for Nextcloud 12. I run Nextcloud 13.0.6 and the issue exists for me. Can anyone confirm?

Steps to reproduce

  1. Upload images (in my case I had about 5gb of JPEGs)
  2. Let previews be generated (e.g. using the preview generator app and a cron job)
  3. Delete those images
  4. Purge trashbin
    (5. Wait for trashbin_retention_obligation is exceeded)

Expected behaviour

The cached previews should be deleted at some point in time...

Actual behaviour

The cached previews are NOT deleted, in my case I have 13GB of preview images of different sizes

Server configuration

Operating system:
Debian

Web server:
Nginx

Database:
Maria

PHP version:
7.2
Nextcloud version: (see Nextcloud admin page)
13.0.6

Updated from an older Nextcloud/ownCloud or fresh install:
Updated; but previews were generated on NC 13

List of activated apps:

Details ``` - activity: 2.6.1 - admin_audit: 1.3.0 - admin_notifications: 1.0.1 - apporder: 0.5.0 - bookmarks: 0.12.2 - bookmarks_fulltextsearch: 1.0.0 - bruteforcesettings: 1.1.0 - calendar: 1.6.1 - caniupdate: 0.1.3 - contacts: 2.1.5 - dav: 1.4.7 - federatedfilesharing: 1.3.1 - files: 1.8.0 - files_fulltextsearch: 1.0.0 - files_fulltextsearch_tesseract: 1.0.0 - files_pdfviewer: 1.2.1 - files_rightclick: 0.8.4 - files_sharing: 1.5.0 - files_texteditor: 2.5.1 - files_trashbin: 1.3.0 - files_versions: 1.6.0 - files_videoplayer: 1.2.0 - fulltextsearch: 1.0.0 - fulltextsearch_elasticsearch: 1.0.0 - gallery: 18.0.0 - impersonate: 1.0.4 - logreader: 2.0.0 - lookup_server_connector: 1.1.0 - metadata: 0.7.0 - nextcloud_announcements: 1.2.0 - notifications: 2.1.2 - oauth2: 1.1.1 - password_policy: 1.3.0 - previewgenerator: 1.1.0 - provisioning_api: 1.3.0 - serverinfo: 1.3.0 - sharerenamer: 1.5.0 - twofactor_backupcodes: 1.2.3 - twofactor_totp: 1.4.1 - updatenotification: 1.3.0 - workflowengine: 1.3.0 ```

Nextcloud configuration:

Details ``` { "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "xxxxxx", "xxxxxx" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "xxxxxxxxx", "dbtype": "mysql", "version": "13.0.6.1", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "memcache.local": "\\OC\\Memcache\\APCu", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "smtp", "mail_smtpauthtype": "LOGIN", "mail_domain": "***REMOVED SENSITIVE VALUE***", "maintenance": false, "logtimezone": "Europe\/Berlin", "loglevel": 0, "theme": "", "trashbin_retention_obligation": "auto, 7", "updater.release.channel": "production", "mail_smtpauth": 1, "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "587", "mail_smtpsecure": "tls", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***" } ```

Activity

  1. nextcloud-bot
    added
    bug
    on Oct 3, 2018

  2. nextcloud-bot commented on Oct 3, 2018

    @nextcloud-bot
    nextcloud-bot
    on Oct 3, 2018
    Member

    GitMate.io thinks possibly related issues are #5558 (After deleting an (image) file, the preview files remain forever), #9523 (A file completely disappears after deleting from a group folder), #8241 (image), #2413 (Previews for empty files broken), and #3041 (Deleting a preview folder stops writing previews to disk).

  3. rullzer commented on Oct 3, 2018

    @rullzer
    rullzer
    on Oct 3, 2018
    Member

    Can you check if you have an entry in your background jobs: OC\Preview\BackgroundCleanupJob?

    Also note that if you use ajax cron it will only delete 10 at a time (to not run into timeouts).

  4. ho1ger commented on Oct 3, 2018

    @ho1ger
    ho1ger
    on Oct 3, 2018
    Author

    I am using a Linux cron job, not Ajax cron.

    Sorry; how do I access OC\Preview\BackgroundCleanupJob?

  5. ho1ger commented on Oct 3, 2018

    @ho1ger
    ho1ger
    on Oct 3, 2018
    Author

    In the logs I found the following which is related to background Cleanup Jobs:

    {"reqId":"0br9L4RsCRwi9HJhnVr9","level":1,"time":"2018-06-08T10:32:51+02:00","remoteAddr":"xxx","user":"--","app":"updater","method":"GET","url":"\/core\/ajax\/update.php?requesttoken=SvT6GXpQphnGbl9Mg9WdNOUBmJ35bJJ5GLpgNhsC%2FQk%3D%3AJ6yiexIZkHu8I24G6L6sYLBY0tyQAuU6XdE5cllluD0%3D","message":"\\OC\\Repair::step: Repair step: Add preview cleanup background jobs","userAgent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit\/605.1.15 (KHTML, like Gecko) Version\/11.1.1 Safari\/605.1.15","version":"13.0.2.1"}
{"reqId":"JiMmzSdZG0ya4yWgnY5p","level":1,"time":"2018-06-12T07:45:32+02:00","remoteAddr":"","user":"--","app":"updater","method":"--","url":"--","message":"\\OC\\Repair::step: Repair step: Add preview cleanup background jobs","userAgent":"--","version":"13.0.3.2"}
{"reqId":"Vuip1twG4c8HRKSb8r1M","level":1,"time":"2018-08-30T18:40:17+02:00","remoteAddr":"","user":"--","app":"updater","method":"--","url":"--","message":"\\OC\\Repair::step: Repair step: Add preview cleanup background jobs","userAgent":"--","version":"13.0.4.0"}
{"reqId":"3N5R2TG63uE5RtzH9Hh1","level":1,"time":"2018-08-30T18:40:48+02:00","remoteAddr":"xxxxxx","user":"--","app":"updater","method":"GET","url":"\/core\/ajax\/update.php?requesttoken=Y9%2FT7HxGSiDjwrAVSLSFWrJQPYx4Lr9Dd44Qy4F6hmc%3D%3AULXl3R8pPGKRuoRsC4G1PcZ7CcVNVsUkJtdyr9VL1gU%3D","message":"\\OC\\Repair::step: Repair step: Add preview cleanup background jobs","userAgent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit\/605.1.15 (KHTML, like Gecko) Version\/11.1.2 Safari\/605.1.15","version":"13.0.4.0"}

  6. ho1ger commented on Oct 3, 2018

    @ho1ger
    ho1ger
    on Oct 3, 2018
    Author

    Q: What would actually happen if I just delete the entire preview folder? Would the preview generator understand that some needed previews are missing and recreate them?

  7. rullzer commented on Oct 3, 2018

    @rullzer
    rullzer
    on Oct 3, 2018
    Member

    AH sorry. Check the oc_jobs table.

    No it won'tyou need to rescan your appdata folder with the occ command

  8. ho1ger commented on Oct 3, 2018

    @ho1ger
    ho1ger
    on Oct 3, 2018
    Author

    Hope I got that right: select * from oc_jobs where class="OC\Preview\BackgroundCleanupJob"; does not return anything.

  9. skjnldsv
    added
    0. Needs triagePending check for reproducibility or if it fits our roadmap
    on Jun 12, 2019

  10. nachoparker commented on Jun 29, 2019

    @nachoparker
    nachoparker
    on Jun 29, 2019
    Member

    I can confirm this is still an issue in NC15.0.8

  11. Iolaum commented on Jul 1, 2019

    @Iolaum
    Iolaum
    on Jul 1, 2019

    I have experienced the same issue in my installation of NextCloud on a nextcloudbox running nextcloudpi. I believe the problem lied with the preview generator app. When I realized the previews were eating up a huge amound of space I ended up deleting the app and manually removed the offending data.

  12. nachoparker commented on Jul 1, 2019

    @nachoparker
    nachoparker
    on Jul 1, 2019
    Member

    @Lolaum since you are using NCP it is configured now to not make unnecessary previews so it should be faster and take up less space.

    Previews are generated one way or another (either by the pregen app or through browsing). Whatever the case may be when you delete the files the previews remain both in the previews folder and in the database and they are never pruned.

  13. skjnldsv commented on Oct 30, 2019

    @skjnldsv
    skjnldsv
    on Oct 30, 2019
    Member

    @rullzer didn't we fixed this a few releases ago?
    I cannot reproduce the issue. Shall we add a background job to cleanup old invalid files id?

  14. skjnldsv
    added
    needs info
    feature: previews and thumbnails
    on Oct 30, 2019
  16. ghost added
    staleTicket or PR with no recent activity
    on Nov 29, 2019
  17. ghost closed this as completedon Dec 13, 2019

  18. James-Oakley commented on Jan 23, 2024

    @James-Oakley
    James-Oakley
    on Jan 23, 2024

    This issue was the one that came up in searches, so someone please link to a more active issue if there is one. I had found that updating previews was taking a very long time, so I moved the appdata/preview folder to one side, rescanned my files, and then reran the preview generator. It took a while, but got there soon enough. The newly generated preview folder contains 61 MB of data. The one I moved contained 1.1 GB of data. So it seems that the preview folder bloats rather than keeping itself clear of previews of files that no longer exist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    0. Needs triagePending check for reproducibility or if it fits our roadmapbugfeature: previews and thumbnailsneeds infostaleTicket or PR with no recent activity

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions