Skip to content

SWIFT object storage as external storage is broken in Nextcloud11 #5866

Closed
Closed
SWIFT object storage as external storage is broken in Nextcloud11#5866
Labels
0. Needs triagePending check for reproducibility or if it fits our roadmapbugfeature: external storagestaleTicket or PR with no recent activity

Description

@rontrompert
rontrompert
opened on Jul 24, 2017

Steps to reproduce

1.Mount SWIFT object storage as external storage
2.Upload a directory with files and subfolders with files in it using something like rclone. This produces output like:

ron@LLT0024:~$ swift list maersk2
1st_file.txt
2nd_file.txt
another_directory/3rd_file.txt

  1. Look at the web interface

Expected behaviour

The following should be shown:
1st_file.txt
2nd_file.txt
another_directory

Actual behaviour

1st_file.txt
2nd_file.txt

Server configuration

Operating system:
CentOS 7.3.1611
Web server:
apache 2.4.6

Database:
mariadb-5.5
PHP version:
7.0
Nextcloud version: (see Nextcloud admin page)
11
Updated from an older Nextcloud/ownCloud or fresh install:
fresh
Where did you install Nextcloud from:
tarball
Signing status:

Signing status 
Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

No errors found

List of activated apps:

App list 
If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder

Nextcloud configuration:

Config report 
If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder

or 

Insert your config.php content here. 
Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …)

Enabled:

  • activity: 2.4.1
  • comments: 1.1.0
  • dav: 1.1.1
  • federatedfilesharing: 1.1.1
  • federation: 1.1.1
  • files: 1.6.1
  • files_external: 1.1.2
  • files_pdfviewer: 1.0.1
  • files_sharing: 1.1.1
  • files_texteditor: 2.2
  • files_trashbin: 1.1.0
  • files_versions: 1.4.0
  • files_videoplayer: 1.0.0
  • gallery: 16.0.0
  • lookup_server_connector: 1.0.0
  • nextcloud_announcements: 1.0
  • notifications: 1.0.1
  • password_policy: 1.1.0
  • provisioning_api: 1.1.0
  • sharebymail: 1.0.1
  • systemtags: 1.1.3
  • theming: 1.1.1
  • twofactor_backupcodes: 1.0.0
  • updatenotification: 1.1.1
  • workflowengine: 1.1.1

Are you using external storage, if yes which one: local/smb/sftp/...
SWIFT
Are you using encryption: yes/no
No
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No

LDAP configuration (delete this part if not used)

LDAP config 
With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your Nextcloud installation folder

Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Eventually replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser:

Operating system:

Logs

Web server error log

Web server error log 
Insert your webserver log here

Nextcloud log (data/nextcloud.log)

Nextcloud log 
Insert your Nextcloud log here

Browser log

Browser log 
Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

Activity

  1. MorrisJobke commented on Jul 25, 2017

    @MorrisJobke
    MorrisJobke
    on Jul 25, 2017
    Member

    cc @icewind1991

  2. MorrisJobke
    added
    0. Needs triagePending check for reproducibility or if it fits our roadmap
    bug
    feature: external storage
    on Jul 25, 2017

  3. icewind1991 commented on Jul 25, 2017

    @icewind1991
    icewind1991
    on Jul 25, 2017
    Member

    Is it only a problem with files that weren't added trough Nc? can you upload file (and then access those files) trough Nc?

  4. rontrompert commented on Aug 14, 2017

    @rontrompert
    rontrompert
    on Aug 14, 2017
    Author

    An object store does not have a directory tree like structure. You have containers or buckets and you have objects. Nothing more. Suppose you have this on your local file system:

    file1
    file2

    dir1

    • file3
    • file4
    • file5

    dir2

    • file6

    Syncing this with an object store (outside of nextcloud) you should get the following objects in your container:

    file1
    file2
    dir1/file3
    dir1/file4
    dir1/file5
    dir2/file6

    When you couple the SWIFT object store with nextcloud you would only see this in the web interface:

    file1
    file2

    Nothing more.

    When dealing with an object store like SWIFT, a slash in a file name is not something to be worried about. It looks like nextcloud is choking on this.

    When I couple a SWIFT container to nextcloud and create files and folders through the web interface, everything is fine.

    Cheers,

    Ron

  5. rontrompert commented on Aug 14, 2017

    @rontrompert
    rontrompert
    on Aug 14, 2017
    Author

    Basically a structure like this:

    dir1

    • file1

    becomes the object:
    dir1/file1

  6. SURFsara-e-infra commented on Oct 29, 2017

    @SURFsara-e-infra
    SURFsara-e-infra
    on Oct 29, 2017

    This is still not fixed in 12.0.3. When creating a directory with a file in it in NC you see the following object in the object store:

    directory/
    directory/file

    Uploading a directory with a file in it with a tool like rclone you will get:

    directory/file

    and this remains unvisible in NC. However, when I created the directory "directory" in the NC web I/F, I suddenly do see:

    directory/
    directory/file

    What is needed is that when uploading a directory with files in it to an object store that would create object with names like directory/file a zero-byte object is created with "Content Type: httpd/unix-directory" and the name "directory/", otherwise NC would not see the objects inside the directory.

    To solve this on the client side may be difficult but on the server side it may be possible to insert such an object during a file scan if it does not exist already.

  7. SURFsara-e-infra commented on Oct 30, 2017

    @SURFsara-e-infra
    SURFsara-e-infra
    on Oct 30, 2017

    I have done some more investigation. SWIFT has these pseudo-hierarchical folders. Clients fail to implement this consistently.

    Uploading a directory with a file in it (directory/file), you see the following:

    rclone gives:

    directory/file

    but fails to generate the directory/ object.

    duck gives:

    directory
    directory/file

    Where "directory" has the content-type: application/directory.

    Neither of the structures above are picked up by Nextcloud. However when I create zero-byte object "directory/" with a "content-type: httpd/unix-directory" myself everything is fine and I see directories and files pop up in Nextcloud.

    Could Nextcloud "fix" this by assuming that when encountering object names like "directory/file" there should als be a zero-byte object named "directory/" with content-type httpd/unix-directory and create it if it does not exist?

  8. nextcloud-bot
    added
    staleTicket or PR with no recent activity
    on Jun 20, 2018

  9. wiswedel commented on May 27, 2019

    @wiswedel
    wiswedel
    on May 27, 2019
    Contributor

    Is this still happening on 16.0.1?

  10. wiswedel
    added
    0. Needs triagePending check for reproducibility or if it fits our roadmap
    bug
    feature: external storage
    staleTicket or PR with no recent activity
    and removed
    0. Needs triagePending check for reproducibility or if it fits our roadmap
    bug
    feature: external storage
    staleTicket or PR with no recent activity
    on May 27, 2019

  11. MorrisJobke commented on May 27, 2019

    @MorrisJobke
    MorrisJobke
    on May 27, 2019
    Member

    Should work in the meantime.

  12. MorrisJobke
    closed this as completedon May 27, 2019

  13. fthouvenin commented on Oct 26, 2019

    @fthouvenin
    fthouvenin
    on Oct 26, 2019

    This also doesn't work in v17...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    0. Needs triagePending check for reproducibility or if it fits our roadmapbugfeature: external storagestaleTicket or PR with no recent activity

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions