Repository navigation
Corrupted preview *-max.png blocks generation of other previews #7749
Description
Activity
cc @rullzer
@rullzer Yes, the file size of the max is zero for the failed files I had so far.
I already read about all previews being generated from the max (which I think in general is a reasonable thing to do).
This bug also affects the preview generator app in the store (also tries to use the max, if it exists and crashes if the max is corrupt).I think one can prevent this bug from happening by using the preview generator app from the beginning (creates previews in single thread so is less likely to crash, while the on-demand preview generation rapidly runs into the limit of threads possible on the Raspberry Pi and uses almost all of its RAM).
I will try the following workaround for fixing my existing Photo folders:
- Delete all previews from filesystem
- Update filecache for previews using
occ files:scan-app-data - Run
occ preview:generate-allafter setting the sizes to the ones the Gallery needs (32 square and 256 height for desktop and 128 square and 1024 height for mobile, width set to 0 for not generating any previews according to this rule).
Yes the preview generation app uses the same stuff from the server as the other preview routes use.
I'll see if we can catch this bad behaviour by checking the size of the max, and maybe even checking if the file is a valid image (if we try to resize).
I guess the easiest solution would be:
- Check if the maxpreview has a size in https://github.com/nextcloud/server/blob/master/lib/private/Preview/Generator.php#L112
- Convert https://github.com/nextcloud/server/blob/master/lib/private/Preview/Generator.php#L332
- Throw Not found exception instead of invalid argument (better to fail with a 404 I guess)
- Delete the max preview if it is invalid
This will cause that preview request to fail with a 404. But at least it will be fully generated next time it is requested. It should not happen that often anyways ;)
I'll take care of this once we are done with 13.
Sounds like a good plan :)
Sometimes other preview sizes are corrupted too. All of them had 0 size. So I just run cronjob every 30 min. which deletes preview files with 0 size. On next access they are regenerated fine.
find "/path_to_NC_appdata/preview" -size 0 -deleteThanks! :)
@rullzer
I find one previous error in log. Maybe it gives some info. It is caused by lock timeout.Error index Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'UPDATEoc_filecacheSETmtime= ?,storage_mtime=? WHERE (mtime<> ? ORstorage_mtime<> ? ORmtimeIS NULL ORstorage_mtimeIS NULL) ANDfileid= ? ' with params [1515763335, 1515763335, 1515763335, 1515763335, 1535624]: SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction /var/www/web/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php - line 128: Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\DBAL\Driver\PDOException)) /var/www/web/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php - line 855: Doctrine\DBAL\DBALException driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Driver\PDOException), 'UPDATEoc_file...', Array)
/var/www/web/lib/private/DB/Connection.php - line 192: Doctrine\DBAL\Connection->executeQuery('UPDATEoc_file...', Array, Array, NULL) /var/www/web/lib/private/Files/Cache/Cache.php - line 320: OC\DB\Connection->executeQuery('UPDATEoc_file...', Array)
/var/www/web/lib/private/Files/Cache/Updater.php - line 247: OC\Files\Cache\Cache->update(1535624, Array)
/var/www/web/lib/private/Files/Cache/Updater.php - line 137: OC\Files\Cache\Updater->correctParentStorageMtime('appdata_ocvyoyj...')
/var/www/web/lib/private/Files/View.php - line 321: OC\Files\Cache\Updater->update('appdata_ocvyoyj...', 1515763335)
/var/www/web/lib/private/Files/View.php - line 1151: OC\Files\View->writeUpdate(Object(OCA\Files_Trashbin\Storage), 'appdata_ocvyoyj...')
/var/www/web/lib/private/Files/View.php - line 559: OC\Files\View->basicOperation('touch', '/appdata_ocvyoy...', Array, NULL)
/var/www/web/lib/private/Files/Node/Folder.php - line 181: OC\Files\View->touch('/appdata_ocvyoy...')
/var/www/web/lib/private/Files/SimpleFS/SimpleFolder.php - line 83: OC\Files\Node\Folder->newFile('128-128-crop.pn...')
/var/www/web/lib/private/Preview/Generator.php - line 337: OC\Files\SimpleFS\SimpleFolder->newFile('128-128-crop.pn...')
/var/www/web/lib/private/Preview/Generator.php - line 125: OC\Preview\Generator->generatePreview(Object(OC\Files\SimpleFS\SimpleFolder), Object(OC\Files\SimpleFS\SimpleFile), 128, 128, true, 667, 1000)
/var/www/web/lib/private/PreviewManager.php - line 201: OC\Preview\Generator->getPreview(Object(OC\Files\Node\File), 128, 128, true, 'fill', 'image/jpeg')
/var/www/web/core/Controller/PreviewController.php - line 113: OC\PreviewManager->getPreview(Object(OC\Files\Node\File), 96, 96, true, 'fill')
[internal function] OC\Core\Controller\PreviewController->getPreview(Object(OC\Files\Node\File), 96, 96, false, false, 'fill')
/var/www/web/lib/private/AppFramework/Http/Dispatcher.php - line 160: call_user_func_array(Array, Array)
/var/www/web/lib/private/AppFramework/Http/Dispatcher.php - line 90: OC\AppFramework\Http\Dispatcher->executeController(Object(OC\Core\Controller\PreviewController), 'getPreview')
/var/www/web/lib/private/AppFramework/App.php - line 114: OC\AppFramework\Http\Dispatcher->dispatch(Object(OC\Core\Controller\PreviewController), 'getPreview')
/var/www/web/lib/private/AppFramework/Routing/RouteActionHandler.php - line 47: OC\AppFramework\App main('OC\Core\Control...', 'getPreview', Object(OC\AppFramework\DependencyInjection\DIContainer), Array)
[internal function] OC\AppFramework\Routing\RouteActionHandler->__invoke(Array)
/var/www/web/lib/private/Route/Router.php - line 299: call_user_func(Object(OC\AppFramework\Routing\RouteActionHandler), Array)
/var/www/web/lib/base.php - line 1004: OC\Route\Router->match('/core/preview.p...')
/var/www/web/index.php - line 48: OC handleRequest()
{main}
`@rullzer Ok, catch it.
This time preview 128-128-crop.png has size 0. It is not pre-generated preview, but on-demand (from mobile browser I suppose)filecache table has this entry and size is also 0.
NC has error logged ~ at the time of creation of preview:
Error index Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'UPDATE
oc_filecacheSETmtime= ?,storage_mtime=? WHERE (mtime<> ? ORstorage_mtime<> ? ORmtimeIS NULL ORstorage_mtimeIS NULL) ANDfileid= ? ' with params [1516278555, 1516278555, 1516278555, 1516278555, 1478155]: SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction
/var/www/web/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php - line 128: Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\DBAL\Driver\PDOException))
/var/www/web/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php - line 855: Doctrine\DBAL\DBALException driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Driver\PDOException), 'UPDATEoc_file...', Array) /var/www/web/lib/private/DB/Connection.php - line 192: Doctrine\DBAL\Connection->executeQuery('UPDATEoc_file...', Array, Array, NULL)
/var/www/web/lib/private/Files/Cache/Cache.php - line 320: OC\DB\Connection->executeQuery('UPDATE `oc_file...', Array)
/var/www/web/lib/private/Files/Cache/Updater.php - line 247: OC\Files\Cache\Cache->update(1478155, Array)
/var/www/web/lib/private/Files/Cache/Updater.php - line 137: OC\Files\Cache\Updater->correctParentStorageMtime('appdata_ocvyoyj...')
/var/www/web/lib/private/Files/View.php - line 321: OC\Files\Cache\Updater->update('appdata_ocvyoyj...', 1516278555)
/var/www/web/lib/private/Files/View.php - line 1151: OC\Files\View->writeUpdate(Object(OCA\Files_Trashbin\Storage), 'appdata_ocvyoyj...')
/var/www/web/lib/private/Files/View.php - line 559: OC\Files\View->basicOperation('touch', '/appdata_ocvyoy...', Array, NULL)
/var/www/web/lib/private/Files/Node/Folder.php - line 181: OC\Files\View->touch('/appdata_ocvyoy...')
/var/www/web/lib/private/Files/SimpleFS/SimpleFolder.php - line 83: OC\Files\Node\Folder->newFile('128-128-crop.pn...')
/var/www/web/lib/private/Preview/Generator.php - line 337: OC\Files\SimpleFS\SimpleFolder->newFile('128-128-crop.pn...')
/var/www/web/lib/private/Preview/Generator.php - line 125: OC\Preview\Generator->generatePreview(Object(OC\Files\SimpleFS\SimpleFolder), Object(OC\Files\SimpleFS\SimpleFile), 128, 128, true, 667, 1000)
/var/www/web/lib/private/PreviewManager.php - line 201: OC\Preview\Generator->getPreview(Object(OC\Files\Node\File), 128, 128, true, 'fill', 'image/jpeg')
/var/www/web/core/Controller/PreviewController.php - line 113: OC\PreviewManager->getPreview(Object(OC\Files\Node\File), 96, 96, true, 'fill')
[internal function] OC\Core\Controller\PreviewController->getPreview(Object(OC\Files\Node\File), 96, 96, false, false, 'fill')
/var/www/web/lib/private/AppFramework/Http/Dispatcher.php - line 160: call_user_func_array(Array, Array)
/var/www/web/lib/private/AppFramework/Http/Dispatcher.php - line 90: OC\AppFramework\Http\Dispatcher->executeController(Object(OC\Core\Controller\PreviewController), 'getPreview')
/var/www/web/lib/private/AppFramework/App.php - line 114: OC\AppFramework\Http\Dispatcher->dispatch(Object(OC\Core\Controller\PreviewController), 'getPreview')
/var/www/web/lib/private/AppFramework/Routing/RouteActionHandler.php - line 47: OC\AppFramework\App main('OC\Core\Control...', 'getPreview', Object(OC\AppFramework\DependencyInjection\DIContainer), Array)
[internal function] OC\AppFramework\Routing\RouteActionHandler->__invoke(Array)
/var/www/web/lib/private/Route/Router.php - line 299: call_user_func(Object(OC\AppFramework\Routing\RouteActionHandler), Array)
/var/www/web/lib/base.php - line 1004: OC\Route\Router->match('/core/preview.p...')
/var/www/web/index.php - line 48: OC handleRequest()
{main}
Steps to reproduce
Expected behaviour
When visiting the folder again and that broken preview is detected (e.g. Gallery app throws "Exception: Preview generation has failed"), the broken file should be regenerated to not block generation of other previews.
Actual behaviour
The software seems to assume the only cause for the broken preview can be a broken file in the first place, so the broken preview file blocks further preview generation. Manually removing the corrupted preview file and running
occ files:scan-app-datato update the cache enables new preview generation at next folder visit.Server configuration
Operating system: Arch Linux ARM (on Raspberry Pi 3)
Web server: Apache 2.4.29
Database: MariaDB 10.1.30
PHP version: 7.0.26
Nextcloud version: 12.0.4
Updated from an older Nextcloud/ownCloud or fresh install: fresh install
Where did you install Nextcloud from: Arch Repositories
Signing status:
Signing status
List of activated apps:
App list
Nextcloud configuration:
Config report
Are you using external storage, if yes which one: no
Are you using encryption: no
Are you using an external user-backend, if yes which one: no
Client configuration
Browser: Opera (Also tested on Firefox)
Operating system: Arch Linux, Windows 10, Linux Mint 17.3
Logs
Nextcloud log (data/nextcloud.log)
Nextcloud log