Nextcloud and S3

Nextcloud version: 12.0.0.3

Nextcloud is connected to S3. When copying files using aws s3 php sdk in a directory bucket,Nextcloud does not see them. But if you look through the content of S3 Browser, everything is there. If the files to burn through nextcloud, no problem. What could be the problem? Some kind of analogue occ filesystem_external:scan ?

ps: Premissions is full.
ps2: Directory and files visible if create path with step by step
league\flysystem adapter s3

$fs_s3->createDir("../dir1");
$fs_s3->createDir("../dir1/subdir2");
$fs_s3->createDir("../dir1/subdir2/subdir3/");
$fs_s3->write("../dir1/subdir2/subdir3/test.txt");
all well and nextcloud displays file

if create file as $fs_s3->write("../dir2/subdir5/subdir6/test.txt")
the file is created but nextcloud does not display (s3 browser sees it)

So you are uploading files directly to S3 and not through Nextcloud?

If so, in the options of the external storage definition there is a dropdown with different options:

Options "Check for changes" set.
also php function aws s3 sdk $cont=$fs_s3->ListContents('',true); work fine and return list of items.

when NC refers to the directory in ceph radosgw log shows that reading the contents of the directory is, but for some reason, NC doesn't display them

For test: added
file_put_contents ('dir to file test',print_r($object,true),FILE_APPEND); 240 line opendir() function
/apps/file_external/lib/Lib/Storage/AmazonS3.php.
result - directory present. NC doesn't display them.

Array
(
    [Key] => file1.txt
    [LastModified] => 2017-10-26T05:34:39.735Z
    [ETag] => "355884b5c38c11828b51aafb7c543581"
    [Size] => 8
    [StorageClass] => STANDARD
    [Owner] => Array
        (
            [ID] => rados
            [DisplayName] => S3 cloud
        )

)
Array
(
    [Key] => test_20171026093814.txt
    [LastModified] => 2017-10-26T05:38:14.789Z
    [ETag] => "23a398553ab3b06032dc7badcca0eaf4"
    [Size] => 3508
    [StorageClass] => STANDARD
    [Owner] => Array
        (
            [ID] => rados
            [DisplayName] => S3 cloud
        )

)
Array
(
    [Prefix] => dir4/
)
Array
(
    [Prefix] => test1/
)
Array
(
    [Prefix] => test2/
)

dir4 and subdir write external scripts

for some reason, the function filetype($path) AmazonS3.php return false on 'dir4'
$this->getConnection()->doesObjectExist($this->bucket,$path.'/') return false, but true for subdir and dir created with NC

NC log:
Debug OC\Files\Cach.. !!! Path 'dir4' is not accessible or present !!!
Why?

If use code from league\flysystem AwS3Adapter.php to determine directory -

 protected function doesDirectoryExist($location)
    {
        // Maybe this isn't an actual key, but a prefix.
        // Do a prefix listing of objects to determine.
        $command = $this->s3Client->getCommand(
            'listObjects',
            [
                'Bucket' => $this->bucket,
                'Prefix' => rtrim($location, '/') . '/',
                'MaxKeys' => 1,
            ]
        );

        try {
            $result = $this->s3Client->execute($command);

            return $result['Contents'] || $result['CommonPrefixes'];
        } catch (S3Exception $e) {
            if ($e->getStatusCode() === 403) {
                return false;
            }

            throw $e;
        }
    }

and change AmazonS3.php function filetype (file/dir)
work is ok.

@nickvergessen

I'm having the exact same issue (NC 12.0.3). I do have the Check for changes: Once every direct access like @nickvergessen mentioned, which makes access much slower, but I do get to see the files/dirs I create straight in S3.

However, a whole tree that I synced (i.e. aws s3 sync ...) from a different bucket goes unnoticed in NC.

If I add new files to one of those ignored folders, nothing changes.
If I move one of those folders under another one previously created from S3 (and present in NC), still not visible.
If I copy files inside one of the invisible folders into a visible one, they show up fine.

@ldapomni I've spotted the filetype() function in file /var/www/nextcloud/apps/files_external/lib/Lib/Storage/AmazonS3.php

288         public function filetype($path) {
289                 $path = $this->normalizePath($path);
290 
291                 if ($this->isRoot($path)) {
292                         return 'dir';
293                 }
294 
295                 try {
296                         if ($this->getConnection()->doesObjectExist($this->bucket, $path)) {
297                                 return 'file';
298                         }
299                         if ($this->getConnection()->doesObjectExist($this->bucket, $path.'/')) {
300                                 return 'dir';
301                         }
302                 } catch (S3Exception $e) {
303                         \OCP\Util::logException('files_external', $e);
304                         return false;
305                 }
306 
307                 return false;
308         }

However, I'm not sure which changes I should made based on your prior comment to test. Also, how to debug this further and gather the needed info?

@icewind1991

cc @icewind1991

I can't reproduce the problem with master (to be NC13) and amazon S3, when creating a folder and adding a file to it trough the aws console, and running occ files:scan it properly detects the newly added folder and file.

Can anyone experiencing this problem try either the lastest NC13 beta (as always when using beta software: backup) or 12.0.4 which should include the same s3 changes.

@sporykhin

After update to NC13, problem still exists.
@sporykhin some error after occ files:scan.

Same problem here, I work normally with S3 folder/files created through NC or directly via AWS S3 GUI, but not with files uploaded with the aws cli. I believe it's somehow connected to the error I keep seeing in the log:

I have a very similar error when trying to delete a folder from an S3 bucket within NC.

Invalid argument supplied for foreach() at /snap/nextcloud/5132/htdocs/apps/files_external/lib/Lib/Storage/AmazonS3.php#291
[Delete] is missing and is a required parameter

Nextcloud 12.0.5


"Exception": "InvalidArgumentException", "Message": "Found 1 error while validating the input provided for the DeleteObjects operation:\\n[Delete] is missing and is a required parameter",
#0 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/Middleware.php(79): AwsApiValidator->validate('DeleteObjects', Object(AwsApiStructureShape), Array)
#1 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php(421): AwsMiddleware::Aws{closure}(*** sensitive parameters replaced ***)
#2 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php(444): AwsS3S3Client::AwsS3{closure}(*** sensitive parameters replaced ***)
#3 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php(378): AwsS3S3Client::AwsS3{closure}(*** sensitive parameters replaced ***)
#4 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php(397): AwsS3S3Client::AwsS3{closure}(*** sensitive parameters replaced ***)
#5 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/Middleware.php(55): AwsS3S3Client::AwsS3{closure}(*** sensitive parameters replaced ***)
#6 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/SSECMiddleware.php(59): AwsMiddleware::Aws{closure}(*** sensitive parameters replaced ***)
#7 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/IdempotencyTokenMiddleware.php(77): AwsS3SSECMiddleware->__invoke(Object(AwsCommand), NULL)
#8 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php(65): AwsIdempotencyTokenMiddleware->__invoke(Object(AwsCommand))
#9 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php(59): AwsAwsClient->executeAsync(Object(AwsCommand))
#10 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php(78): AwsAwsClient->execute(Object(AwsCommand))
#11 /snap/nextcloud/5132/htdocs/apps/files_external/lib/Lib/Storage/AmazonS3.php(255): AwsAwsClient->__call('deleteObjects', Array)
#12 /snap/nextcloud/5132/htdocs/apps/files_external/lib/Lib/Storage/AmazonS3.php(224): OCAFiles_ExternalLibStorageAmazonS3->batchDelete('2018/02')
#13 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(87): OCAFiles_ExternalLibStorageAmazonS3->rmdir('2018/02')
#14 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/PermissionsMask.php(106): OCFilesStorageWrapperWrapper->rmdir('2018/02')
#15 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(87): OCFilesStorageWrapperPermissionsMask->rmdir('2018/02')
#16 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Availability.php(95): OCFilesStorageWrapperWrapper->rmdir('2018/02')
#17 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Encryption.php(298): OCFilesStorageWrapperAvailability->rmdir('2018/02')
#18 [internal function]: OCFilesStorageWrapperEncryption->rmdir('2018/02')
#19 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Storage.php(258): call_user_func_array(Array, Array)
#20 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Storage.php(189): OCAFiles_TrashbinStorage->doDelete('2018/02', 'rmdir')
#21 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Encryption.php(617): OCAFiles_TrashbinStorage->rmdir('2018/02')
#22 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(569): OCFilesStorageWrapperEncryption->moveFromStorage(Object(OCAFiles_TrashbinStorage), '2018/02', 'files_trashbin/...')
#23 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Trashbin.php(251): OCFilesStorageWrapperWrapper->moveFromStorage(Object(OCAFiles_TrashbinStorage), '2018/02', 'files_trashbin/...')
#24 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Storage.php(247): OCAFiles_TrashbinTrashbin::move2trash('AWS/2018/02', false)
#25 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Storage.php(189): OCAFiles_TrashbinStorage->doDelete('2018/02', 'rmdir')
#26 /snap/nextcloud/5132/htdocs/lib/private/Files/View.php(1159): OCAFiles_TrashbinStorage->rmdir('2018/02')
#27 /snap/nextcloud/5132/htdocs/lib/private/Files/View.php(348): OCFilesView->basicOperation('rmdir', '/AWS/2018/02', Array)
#28 /snap/nextcloud/5132/htdocs/apps/dav/lib/Connector/Sabre/Directory.php(303): OCFilesView->rmdir('/AWS/2018/02')
#29 /snap/nextcloud/5132/htdocs/3rdparty/sabre/dav/lib/DAV/Tree.php(179): OCADAVConnectorSabreDirectory->delete()
#30 /snap/nextcloud/5132/htdocs/3rdparty/sabre/dav/lib/DAV/CorePlugin.php(287): SabreDAVTree->delete('AWS/2018/02')
#31 [internal function]: SabreDAVCorePlugin->httpDelete(Object(SabreHTTPRequest), Object(SabreHTTPResponse))
#32 /snap/nextcloud/5132/htdocs/3rdparty/sabre/event/lib/EventEmitterTrait.php(105): call_user_func_array(Array, Array)
#33 /snap/nextcloud/5132/htdocs/3rdparty/sabre/dav/lib/DAV/Server.php(479): SabreEventEventEmitter->emit('method:DELETE', Array)
#34 /snap/nextcloud/5132/htdocs/3rdparty/sabre/dav/lib/DAV/Server.php(254): SabreDAVServer->invokeMethod(Object(SabreHTTPRequest), Object(SabreHTTPResponse))
#35 /snap/nextcloud/5132/htdocs/apps/dav/appinfo/v1/webdav.php(76): SabreDAVServer->exec()
#36 /snap/nextcloud/5132/htdocs/remote.php(162): require_once('/snap/nextcloud...')
#37 {main}
"File": "/snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/Api/Validator.php", "Line": 65

Off topic: S3 external storages are basically unusable, even when they're working. They're just too slow.

I had the same issue with S3 DigitalOcean Spaces & Nextcloud, so to make Nextcloud efficient again, I had to switch to DO Block Storage.

Unfortunately, I don't have a choice in the platform I use. Two weeks ago my client was mandated to use Heroku-Shield in order to meet HIPPA Compliance. Since Heroku uses a ephemeral disk platform, S3 was our only storage option that met HIPPA Compliancy for our client.

@kesselb

@kesselb

I think the cleanest way is to write the files to s3 through nextcloud. In this case you can use scan never.

Tried copying relatively large dirs from local disk to S3 with nextcloud. Half of folders are not visible, although copied fine. Need to enable rescan again.

So it's not a solution...

@dimm0

@dimm0 This might be something for another issue. The case here with the directory support will be fixed soon.

@sbw Sounds good to me. Would you mind to create a new enhancement issue for this? (I think it should be like that maybe something broken).

I applied patch https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3 and Fixed my Issue. I Running Nextcloud 15.0.7

Thank you

still broken in 16.0.1 it seems but the patch still fixes it :(

I can confirm the patch mentioned above resolved the issue for me. I'm running nextcloud 16.0.1 and using Digital Ocean spaces for external storage, with folders/files populated using rclone directly to DO spaces. I basically just downloaded the modified AmazonS3.php file into apps/files_external/lib/Lib/Storage/ and then re-ran php occ files:scan <user> --path=/<user>/files/<name of the external folder>. I think php occ files:scan --all might have worked as well though. Before doing that I had the same issue as many others, where none of the directories populated by rclone were showing up in nextcloud.

@kesselb

@kesselb wrote, "@sbw Sounds good to me. Would you mind to create a new enhancement issue for this? (I think it should be like that maybe something broken)."

Sorry, I can't tell whether that pertains to the original issue (S3 external storage stuff never shows up in Nextcloud) or my off-topic observation that "S3 external storages are basically unusable, even when they're working. They're just too slow."

I hope it's the latter. I hope you're suggesting I create an "enhancement issue" to make S3 external storage fast enough to be usable. If that's what you're suggesting, I'll do that when I have time. (It's not an enhancement, really, but a defect, in my opinion: I believe Nextcloud is meant to immediately display the information it already has from a previous scan. I don't know what Nextcloud is doing instead, but whatever it is, it can't be the intended behavior.)

@alexklibisz

hey @alexklibisz i was trying to follow what you did but I couldn't modify or download updated AmazonS3.php due to read-only file system. How did you bypass it?

@sbw

S3 external storages are basically unusable, even when they're working. They're just too slow.

@sbw this one.

hey @alexklibisz i was trying to follow what you did but I couldn't modify or download updated AmazonS3.php due to read-only file system. How did you bypass it?

@hasangokdag If you are using the snap package applying this patch is not possible.

I am running nextcloud in a docker container, so I have root access inside of it by default.

…

On Sat, Jun 22, 2019, 06:42 Daniel Kesselberg ***@***.***> wrote: S3 external storages are basically unusable, even when they're working. They're just too slow. @sbw <https://github.com/sbw> this one. hey @alexklibisz <https://github.com/alexklibisz> i was trying to follow what you did but I couldn't modify or download updated AmazonS3.php due to read-only file system. How did you bypass it? @hasangokdag <https://github.com/hasangokdag> If you are using the snap package applying this patch is not possible. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6954?email_source=notifications&email_token=AB5E27FS73YARJP7HA6SH2LP3X6Z7A5CNFSM4EAXSQC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYKGWDA#issuecomment-504654604>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB5E27DY7ANRGFACFPB4YRTP3X6Z7ANCNFSM4EAXSQCQ> .

For those like me using snap and impatient, you can upgrade to Nextcloud 17:

sudo snap refresh nextcloud --channel=17/edge
sudo snap disable nextcloud && sudo snap enable nextcloud

Navigation Menu

Repository navigation

Nextcloud and S3 #6954

32 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Nextcloud and S3 #6954

Description

Activity

ldapomni commented on Oct 25, 2017

nickvergessen commented on Oct 25, 2017

ldapomni commented on Oct 26, 2017

ldapomni commented on Oct 26, 2017

ldapomni commented on Oct 26, 2017

ldapomni commented on Oct 26, 2017

default50 commented on Nov 19, 2017

MorrisJobke commented on Dec 18, 2017

icewind1991 commented on Dec 19, 2017

ldapomni commented on Feb 14, 2018

velimirg commented on Feb 21, 2018

kruffin commented on Mar 9, 2018

32 remaining items

HLFH commented on Apr 14, 2019

wanchic commented on Apr 15, 2019

dimm0 commented on Apr 16, 2019

kesselb commented on Apr 16, 2019

JarbyDev commented on May 13, 2019

imxLucid commented on May 21, 2019

alexklibisz commented on May 27, 2019

sbw commented on May 30, 2019

hasangokdag commented on Jun 22, 2019

kesselb commented on Jun 22, 2019

alexklibisz commented on Jun 22, 2019

crypdick commented on Oct 31, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions