Repository navigation
Minio Issue with S3 external storage module #9808
Description
Activity
I can corroborate this problem (or a version at least)
Nextcloud: Docker nextcloud:latest image == 13.0.4-apache
Minio: Docker minio/minio:2018-06-09T03:43:35Z imageMinio is added to Nextcloud as an S3 external source -- not as primary storage -- connecting to an existing bucket with an existing directory tree of data.
Minio is serving a disk from a local computer, so I can access the underlying file structure multiple ways (at console, through Minio, and through Nextcloud).
As we start, within the bucket:
Minio sees:
2011/10/data.csv
Nextcloud sees:
(no files in bucket)
If I make a directory "test" using Nextcloud:
Minio sees:
2011/10/data.csv test/Nextcloud sees:
test/If I make a directory "test" within "test" using Nextcloud:
Minio sees:
2011/10/data.csv
test/test/Nextcloud sees:
(no files in bucket)
[ then destroy test/ and test/test/ directory ]
If I then make a directory "test/", Nextcloud can see the folder, as before.
If I then upload a file "test/foo.txt" , Nextcloud can no longer see the folder "test/" -- but the file does exist on the filesystem and in minio
If I then remove test/foo.txt (using Minio), Nextcloud can now see the folder "test/" again.
At a high level, it appears the Nextcloud "no longer sees" top-level folders once they have content (a subdir or a file). We can't test if this is true for subdirs because we cannot create subdirs w/o losing the top-level files.
Reacted by Thomas TanghusSimilar issue here. I am unable to create or move folders to external storage (Minio S3). I can, however, create, upload and move files. That means I can put files to the root of external storage and not make any folders. Even the folders that do already exist are not visible in Nextcloud.
Similar issue here.
There are lots of logs like
Debug | OC\Files\Cache\Scanner | !!! Path 'test' is not accessible or present !!!.By looking for the minio's access log, I found nextcloud server try to list dir on server(?). But minio returns 404.
logs here.
54.64.***.* - - [28/Oct/2018:04:17:14 +0100] "HEAD /test HTTP/1.1" 404 0 "-" "aws-sdk-php/3.62.10 GuzzleHttp/6.2.1 curl/7.61.1 PHP/7.2.10" 54.64.***.* - - [28/Oct/2018:04:17:14 +0100] "HEAD /test/ HTTP/1.1" 404 0 "-" "aws-sdk-php/3.62.10 GuzzleHttp/6.2.1 curl/7.61.1 PHP/7.2.10"Probably that this is caused by minio/minio#4434
but
won't fix😢I were also running into smaller issues with directories in directories.
Could you try this one?
https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3 (patch file)#14913 (patch file)Reacted by Paul Pettigrew and DReacted by Paul PettigrewI can confirm that we have manually applied kesselb scan-external-s3 patch and it has solved this issue. We are running Nextcloud version 15.0.4
Please consider accepting patch into upstream so arrives in next Stable version, thank you.
Paul
Reacted by Paul Pettigrew and DReacted by Paul PettigrewThe following #9808 (comment) gives error
error: unrecognized inputReproducable
>root@nextcloud:/var/www/html# curl -L https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3.patch -o /tmp/scan-external-s3.patch % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 273k 0 273k 0 0 507k 0 --:--:-- --:--:-- --:--:-- 507k >root@nextcloud:/var/www/html# git apply /tmp/scan-external-s3.patch error: unrecognized input
Not sure if this quite the right repo so apologies if it needs to be in one of the others although I couldn't see one that looked like it would be more appropriate.
It used to the case that It was possible to use a Minio server with the external storage module but it looks like since an update to Minio nextcloud can now only see files in the root of the bucket.
I already raised it with the minio project, but I thought I'd also raise it here incase there's something that can be done from the nextcloud end,
I know there's not really folders in S3 but most tools (including nextcloud) still visualise the structure as files/folders
minio/minio#5916
Steps to reproduce
Expected behaviour
I would expect to see everything within the bucket that nextcloud has permissions to see
Actual behaviour
Files in the root of the bucket are visble, everything else isn't
Server configuration
Ubuntu 18.04,
Apache 2.4.29
MariaDB 10.1.29-MariaDB-6
PHP 7.2.5-0ubuntu0.18.04.1
13.02
Fresh install (I think, but it might have been upgraded from 13.01)
Nextcloud site
Signing status:
No errors have been found.
List of activated apps:
App list
Enabled:
Disabled:
Are you using external storage, if yes which one:
as default no
primary use has a folder that is s3 via minio configured as "external storage"
Are you using encryption:
no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no
LDAP configuration (delete this part if not used)
Client configuration
Browser:
Multiple
Operating system:
Multiple