Repository navigation
Performance issues with WebDav (remote.php) #7964
Description
Steps to reproduce
- Add WebDav network drive using nextcloud remote.php
Big file:
- copy a large file to nextcloud using this drive
Small files:
- Move a large amount from one folder to another
Expected behaviour
Big file:
Upload of the file shouldn't hang at the last 1% of the upload.
Small files:
Moving and uploading should be quick and not with a pause between every file
Actual behaviour
Big file:
Upload hangs at 99% for a couple of minutes, then it finishes or produces an error "Element not found".
First 99% upload speed is at about 200mb/s and done in about 10 seconds.
Small files:
Upload and moving takes ages.
Server configuration
Operating system:
Debian 8.10 as Host, Alpine Linux Docker image with nextcloud.
MySql is hosted by Host.
Web server:
Nginx 1.12.2-rl
Database:
MySQL 5.5.58
PHP version:
7.1.9
Nextcloud version: (see Nextcloud admin page)
12.0.4
Updated from an older Nextcloud/ownCloud or fresh install:
Fresh install
Where did you install Nextcloud from:
Docker image from linuxserver.io
Signing status:
Signing status
No errors have been found.
List of activated apps:
App list
Enabled:
- activity: 2.5.2
- audioplayer: 2.2.2
- bookmarks: 0.10.1
- bruteforcesettings: 1.0.3
- calendar: 1.5.7
- comments: 1.2.0
- contacts: 2.0.1
- dav: 1.3.0
- drawio: 0.8.8
- external: 2.0.3
- federatedfilesharing: 1.2.0
- federation: 1.2.0
- files: 1.7.2
- files_pdfviewer: 1.1.1
- files_sharing: 1.4.0
- files_texteditor: 2.4.1
- files_trashbin: 1.2.0
- files_versions: 1.5.0
- files_videoplayer: 1.1.0
- firstrunwizard: 2.1
- gallery: 17.0.0
- groupfolders: 1.1.0
- keeweb: 0.4.0
- logreader: 2.0.0
- lookup_server_connector: 1.0.0
- nextcloud_announcements: 1.1
- notifications: 2.0.0
- oauth2: 1.0.5
- password_policy: 1.2.2
- provisioning_api: 1.2.0
- quota_warning: 1.1.1
- ransomware_protection: 1.0.5
- serverinfo: 1.2.0
- sharebymail: 1.2.0
- survey_client: 1.0.0
- systemtags: 1.2.0
- tasks: 0.9.5
- theming: 1.3.0
- twofactor_backupcodes: 1.1.1
- updatenotification: 1.2.0
- workflowengine: 1.2.0
Disabled:
- admin_audit
- encryption
- files_external
- ownbackup
- user_external
- user_ldap
Nextcloud configuration:
Config report
{
"system": {
"memcache.local": "\\OC\\Memcache\\APCu",
"datadirectory": "\/data",
"instanceid": "oc3e67um6t92",
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
"***REMOVED SENSITIVE VALUE***",
"***REMOVED SENSITIVE VALUE***"
],
"overwrite.cli.url": "https:\/\/"***REMOVED SENSITIVE VALUE***"",
"dbtype": "mysql",
"version": "12.0.4.3",
"dbname": "nextcloud",
"dbhost": "192.168.178.100:3306",
"dbport": "",
"dbtableprefix": "oc_",
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"installed": true,
"maintenance": false,
"auth.bruteforce.protection.enabled": false
},
"apps": {
"activity": {
"enabled": "yes",
"installed_version": "2.5.2",
"types": "filesystem"
},
"audioplayer": {
"enabled": "yes",
"installed_version": "2.2.2",
"types": "filesystem"
},
"backgroundjob": {
"lastjob": "56"
},
"bookmarks": {
"enabled": "yes",
"installed_version": "0.10.1",
"types": ""
},
"bruteforcesettings": {
"enabled": "yes",
"installed_version": "1.0.3",
"types": ""
},
"calendar": {
"enabled": "yes",
"installed_version": "1.5.7",
"types": ""
},
"comments": {
"enabled": "yes",
"installed_version": "1.2.0",
"types": "logging"
},
"contacts": {
"enabled": "yes",
"installed_version": "2.0.1",
"types": ""
},
"core": {
"backgroundjobs_mode": "cron",
"installed.bundles": "[\"CoreBundle\"]",
"installedat": "1515456733.5354",
"lastcron": "1516376717",
"lastupdateResult": "[]",
"lastupdatedat": "1516376551",
"oc.integritycheck.checker": "[]",
"public_files": "files_sharing\/public.php",
"public_webdav": "dav\/appinfo\/v1\/publicwebdav.php",
"scss.variables": "d90e2783d0e072805f2aa28dc1097658",
"vendor": "nextcloud"
},
"dav": {
"enabled": "yes",
"installed_version": "1.3.0",
"types": "filesystem"
},
"drawio": {
"enabled": "yes",
"installed_version": "0.8.8",
"types": "filesystem"
},
"external": {
"enabled": "yes",
"installed_version": "2.0.3",
"types": ""
},
"federatedfilesharing": {
"enabled": "yes",
"installed_version": "1.2.0",
"types": ""
},
"federation": {
"enabled": "yes",
"installed_version": "1.2.0",
"types": "authentication"
},
"files": {
"cronjob_scan_files": "500",
"enabled": "yes",
"installed_version": "1.7.2",
"types": "filesystem"
},
"files_pdfviewer": {
"enabled": "yes",
"installed_version": "1.1.1",
"ocsid": "166049",
"types": ""
},
"files_sharing": {
"enabled": "yes",
"installed_version": "1.4.0",
"types": "filesystem"
},
"files_texteditor": {
"enabled": "yes",
"installed_version": "2.4.1",
"ocsid": "166051",
"types": ""
},
"files_trashbin": {
"enabled": "yes",
"installed_version": "1.2.0",
"types": "filesystem"
},
"files_versions": {
"enabled": "yes",
"installed_version": "1.5.0",
"types": "filesystem"
},
"files_videoplayer": {
"enabled": "yes",
"installed_version": "1.1.0",
"types": ""
},
"firstrunwizard": {
"enabled": "yes",
"installed_version": "2.1",
"types": "logging"
},
"gallery": {
"enabled": "yes",
"installed_version": "17.0.0",
"types": ""
},
"groupfolders": {
"enabled": "yes",
"installed_version": "1.1.0",
"types": "filesystem"
},
"keeweb": {
"enabled": "yes",
"installed_version": "0.4.0",
"types": ""
},
"logreader": {
"enabled": "yes",
"installed_version": "2.0.0",
"ocsid": "170871",
"relativedates": "1",
"types": ""
},
"lookup_server_connector": {
"enabled": "yes",
"installed_version": "1.0.0",
"types": "authentication"
},
"music": {
"enabled": "no",
"installed_version": "0.5.5",
"types": "filesystem"
},
"nextcloud_announcements": {
"enabled": "yes",
"installed_version": "1.1",
"pub_date": "Sat, 10 Dec 2016 00:00:00 +0100",
"types": "logging"
},
"notifications": {
"enabled": "yes",
"installed_version": "2.0.0",
"types": "logging"
},
"oauth2": {
"enabled": "yes",
"installed_version": "1.0.5",
"types": "authentication"
},
"ocdownloader": {
"enabled": "no",
"installed_version": "1.5.5",
"types": ""
},
"ownbackup": {
"enabled": "no",
"installed_version": "17.5.0",
"types": ""
},
"password_policy": {
"enabled": "yes",
"installed_version": "1.2.2",
"types": ""
},
"provisioning_api": {
"enabled": "yes",
"installed_version": "1.2.0",
"types": "prevent_group_restriction"
},
"quota_warning": {
"enabled": "yes",
"installed_version": "1.1.1",
"types": "filesystem"
},
"ransomware_protection": {
"enabled": "yes",
"installed_version": "1.0.5",
"types": "logging"
},
"serverinfo": {
"enabled": "yes",
"installed_version": "1.2.0",
"types": ""
},
"sharebymail": {
"enabled": "yes",
"installed_version": "1.2.0",
"types": "filesystem"
},
"survey_client": {
"enabled": "yes",
"installed_version": "1.0.0",
"types": ""
},
"systemtags": {
"enabled": "yes",
"installed_version": "1.2.0",
"types": "logging"
},
"tasks": {
"enabled": "yes",
"installed_version": "0.9.5",
"types": ""
},
"theming": {
"cachebuster": "2",
"enabled": "yes",
"installed_version": "1.3.0",
"name": "DutchCloud",
"types": "logging",
"url": "https:\/\/dutches.ddns.net"
},
"twofactor_backupcodes": {
"enabled": "yes",
"installed_version": "1.1.1",
"types": ""
},
"updatenotification": {
"audioplayer": "2.2.3",
"enabled": "yes",
"installed_version": "1.2.0",
"types": "",
"update_check_errors": "0"
},
"workflowengine": {
"enabled": "yes",
"installed_version": "1.2.0",
"types": "filesystem"
}
}
}
Are you using external storage, if yes which one: local/smb/sftp/...
No
Are you using encryption: yes/no
No
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No
Client configuration
Browser:
Using WebDav.
Operating system:
Windows 10
Logs
Web server error log
Web server error log
Error log is empty.
Nextcloud log (data/nextcloud.log)
Nextcloud log
No errors.
Activity
Flying--Dutchman commentedon Jan 19, 2018
It looks like nextcloud does alot of stuff in the background, when the actually 'upload' is done.
There is a pause after each file, which increases depending on the size of the uploaded file.
Why does the action after uploading a ~500mb file take multiple minutes?
Neither CPU nor RAM is at full capacity.
Copy process:
CPU and RAM usage of the server:
nickvergessen commentedon Jan 22, 2018
Well check the Disk IO, after the upload is "finished" the uploaded chunks need to be assembled to the complete file again.
Flying--Dutchman commentedon Jan 22, 2018
Correct, the nginx worker writes with 1.500kb to the disk after the upload. Does Nextcloud have to wait for this operation? Can't it be done in the background, asynchronously?
MorrisJobke commentedon Mar 1, 2018
Sadly not, otherwise the request can't return with a success code (Something like "Created"), but this is required by webdav. We would need to say: "yes we received it, but it is in an intermediate state", which is not available in webdav :/ So this needs to be done synchronously. Sorry for the bad news here.