S3: High connectivity check rate

Steps to reproduce

Upload a file
List the file several times
Get S3 Bucket log
Count amount of HEAD requests to /

Expected behaviour

These connectivity checks would be cached and reused

Actual behaviour

The connectivity checks are done once every transaction

Server configuration detail

Operating system: Linux 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64

Webserver: nginx/1.16.0 (fpm-fcgi)

Database: pgsql PostgreSQL 11.4 (Debian 11.4-1.pgdg90+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516, 64-bit

PHP version:

7.3.6
Modules loaded: Core, date, libxml, pcre, zlib, filter, hash, readline, Reflection, SPL, session, cgi-fcgi, bz2, ctype, curl, dom, fileinfo, ftp, gd, gmp, iconv, intl, json, ldap, mbstring, openssl, pcntl, PDO, pgsql, posix, standard, SimpleXML, smbclient, sodium, sqlite3, xml, xmlwriter, zip, exif, imap, mysqlnd, pdo_pgsql, pdo_sqlite, Phar, xmlreader, pdo_mysql, apcu, igbinary, redis, memcached, imagick, mcrypt, libsmbclient, Zend OPcache

Nextcloud version: 16.0.3 - 16.0.3.0

Updated from an older Nextcloud/ownCloud or fresh install: Updated from older version

Where did you install Nextcloud from: Docker Hub (official image) (edited from linuxserver/nextcloud, still persists on official image)

Signing status

)

List of activated apps

Enabled:- accessibility: 1.2.0- activity: 2.9.1- audioplayer: 2.7.2- bookmarks: 1.0.6- bruteforcesettings: 1.4.0- calendar: 1.7.0- cloud_federation_api: 0.2.0- comments: 1.6.0- dav: 1.9.2- deck: 0.6.4- federatedfilesharing: 1.6.0- federation: 1.6.0- files: 1.11.0- files_external: 1.7.0- files_markdown: 2.0.6- files_pdfviewer: 1.5.0- files_rightclick: 0.13.0- files_sharing: 1.8.0- files_texteditor: 2.8.0- files_trashbin: 1.6.0- files_versions: 1.9.0- files_videoplayer: 1.5.0- firstrunwizard: 2.5.0- groupfolders: 4.0.3- issuetemplate: 0.5.0- logreader: 2.1.0- lookup_server_connector: 1.4.0- mail: 0.15.1- metadata: 0.9.0- news: 13.1.6- nextcloud_announcements: 1.5.0- notifications: 2.4.1- oauth2: 1.4.2- onlyoffice: 2.4.0- password_policy: 1.6.0- privacy: 1.0.0- provisioning_api: 1.6.0- recommendations: 0.4.0- registration: 0.4.6- serverinfo: 1.6.0- sharebymail: 1.6.0- spreed: 6.0.3- support: 1.0.0- survey_client: 1.4.0- systemtags: 1.6.0- theming: 1.7.0- twofactor_backupcodes: 1.5.0- twofactor_totp: 2.1.2- updatenotification: 1.6.0- viewer: 1.0.0- workflowengine: 1.6.0
Disabled:- admin_audit- apporder- contacts- encryption- end_to_end_encryption- flowupload- gallery- keeweb- notes- ocdownloader- polls- richdocuments- twofactor_u2f- twofactor_yubikey- user_ldap

Configuration (config/config.php)

{   "memcache.local": "\\OC\\Memcache\\APCu",   "memcache.distributed": "\\OC\\Memcache\\Redis",   "memcache.locking": "\\OC\\Memcache\\Redis",   "datadirectory": "***REMOVED SENSITIVE VALUE***",   "version": "16.0.3.0",   "passwordsalt": "***REMOVED SENSITIVE VALUE***",   "secret": "***REMOVED SENSITIVE VALUE***",   "apps_paths": [       {           "path": "\/config\/www\/nextcloud\/apps",           "url": "\/apps",           "writable": false       },       {           "path": "\/config\/www\/nextcloud\/custom_apps",           "url": "\/custom_apps",           "writable": true       }   ],   "redis": {       "host": "***REMOVED SENSITIVE VALUE***",       "port": 6379   },   "instanceid": "***REMOVED SENSITIVE VALUE***",   "mail_from_address": "***REMOVED SENSITIVE VALUE***",   "trusted_domains": [       "***REMOVED SENSITIVE VALUE***"   ],   "dbtype": "pgsql",   "dbname": "***REMOVED SENSITIVE VALUE***",   "dbhost": "***REMOVED SENSITIVE VALUE***",   "dbport": "",   "dbtableprefix": "oc_",   "dbuser": "***REMOVED SENSITIVE VALUE***",   "dbpassword": "***REMOVED SENSITIVE VALUE***",   "overwrite.cli.url": "https:\/\/cloud.redxen.eu",   "overwriteprotocol": "https",   "trusted_proxies": "***REMOVED SENSITIVE VALUE***",   "installed": true,   "trashbin_retention_obligation": "auto,7",   "enable_previews": false,   "forwarded_for_headers": [       "HTTP_X_FORWARDED_FOR"   ],   "objectstore": {       "class": "OC\\Files\\ObjectStore\\S3",       "arguments": {           "bucket": "rxnc",           "key": "***REMOVED SENSITIVE VALUE***",           "secret": "***REMOVED SENSITIVE VALUE***",           "port": 443,           "use_ssl": true,           "region": "eu-central-1"       }   },   "filesystem_check_changes": 0,   "skeletondirectory": "",   "mail_smtpmode": "smtp",   "mail_sendmailmode": "smtp",   "mail_domain": "***REMOVED SENSITIVE VALUE***",   "mail_smtpauthtype": "LOGIN",   "mail_smtpsecure": "tls",   "mail_smtpauth": 1,   "mail_smtphost": "***REMOVED SENSITIVE VALUE***",   "mail_smtpport": "587",   "mail_smtpname": "***REMOVED SENSITIVE VALUE***",   "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",   "twofactor_enforced": false,   "config_is_read_only": true,   "htaccess.RewriteBase": "\/",   "twofactor_enforced_groups": [       "admin"   ],   "twofactor_enforced_excluded_groups": [],   "theme": "",   "loglevel": 2
}

Are you using external storage, if yes which one: S3

Are you using encryption: false

Are you using an external user-backend, if yes which one:

Client configuration

Browser: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Falkon/3.1.0 Chrome/73.0.3683.105 Safari/537.36

Operating system: Linux RXHome 5.2.4-zen1-1-zen #1 ZEN SMP PREEMPT Sun Jul 28 10:52:59 UTC 2019 x86_64 GNU/Linux

Logs

S3 Snippet

REST.HEAD.BUCKET - "HEAD / HTTP/1.1" 200 - - - 14 13 "-" "aws-sdk-php/3.62.10 GuzzleHttp/6.3.3 curl/7.65.1 PHP/7.3.6" - jvhUHRNP2kS1fT7hAOw7BD3Uo4fEI2GFOmizgDZLTMnClxKFxsvpPEcCkczk8Tbta2vGMj75Bxw= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com - REST.GET.OBJECT urn%253Aoid%253A18610 "GET /urn%3Aoid%3A18610 HTTP/1.0" 200 - 452232 452232 14 13 "-" "aws-sdk-php/3.62.10" - JN4vhq/fxdKhRIVDV5lqdfsHuEPdSJkRoM2Cn4I+0wEm2wqOc9dQjBOT3PiuSGuf+k9+7RpaP5k= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com - REST.HEAD.BUCKET - "HEAD / HTTP/1.1" 200 - - - 13 12 "-" "aws-sdk-php/3.62.10 GuzzleHttp/6.3.3 curl/7.65.1 PHP/7.3.6" - HTVTiYm89K93hK47xepazKX4cxlwFQml+Taas9xekTb+FWZU5URjne9chLbinSUVf8BhrfDF5tc= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com - REST.GET.OBJECT urn%253Aoid%253A18610 "GET /urn%3Aoid%3A18610 HTTP/1.0" 200 - 452232 452232 70 68 "-" "aws-sdk-php/3.62.10" - DNE3WG1DyOS9EkNE7TxLB9f4d8UBCR1cp6z+YVeB37tr4O3tjr3iRS7Qe6FaE01iX6jMxfWc5sc= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com TLSv1.2 REST.HEAD.BUCKET - "HEAD / HTTP/1.1" 200 - - - 1 1 "-" "aws-sdk-php/3.62.10 GuzzleHttp/6.3.3 curl/7.65.1 PHP/7.3.6" - QNONeEhmzY99gpsl+KUuF1NWwHvRofF4XIgw/FNjMA0ObEU90/3K1KTE63BVZDTwltHfo8lOsfM= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com TLSv1.2 REST.GET.OBJECT urn%253Aoid%253A18610 "GET /urn%3Aoid%3A18610 HTTP/1.0" 200 - 452232 452232 9 8 "-" "aws-sdk-php/3.62.10" - dey+MVD3CtzWKdwrzD4f3CGOnsSywAYo81L7XujD+n7Aiet8HT08s9xR5NPfVU6SWpeZH6HosLw= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com TLSv1.2 REST.HEAD.BUCKET - "HEAD / HTTP/1.1" 200 - - - 13 12 "-" "aws-sdk-php/3.62.10 GuzzleHttp/6.3.3 curl/7.65.1 PHP/7.3.6" - /9TTUBj8Th7ktOPHSc4e1Y90fRqWQlaDJwkFRmsIuCd5TH25CBfD5fuPlAATYui7RoISigQkJ34= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com TLSv1.2 REST.GET.OBJECT urn%253Aoid%253A18610 "GET /urn%3Aoid%3A18610 HTTP/1.0" 200 - 452232 452232 14 12 "-" "aws-sdk-php/3.62.10" - p5v7iq5yeKvAO1nHY7AGaSW+gu2Ama3z1PQLavuFRfdy36YQMlR4D/DzappxaZJpYpNcLAJ6Pq4= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com TLSv1.2 REST.HEAD.BUCKET - "HEAD / HTTP/1.1" 200 - - - 11 11 "-" "aws-sdk-php/3.62.10 GuzzleHttp/6.3.3 curl/7.65.1 PHP/7.3.6" - kxTLM3pW3Sr1rhHjFnVv0tYOOgiEHAyZ6916ycg7L/X4c7Hvz+ImIf3vetrDcSYTz69eT3298a4= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com TLSv1.2 REST.GET.OBJECT urn%253Aoid%253A18610 "GET /urn%3Aoid%3A18610 HTTP/1.0" 200 - 452232 452232 22 20 "-" "aws-sdk-php/3.62.10" - FMFKsuDiQUUuiEnmFRhiJDp3OpDLjkRAzQLbYF6g4zuC1dSgeoLD0lCMKTVilvSZhJZEAa9B7m4= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com TLSv1.2 REST.HEAD.BUCKET - "HEAD / HTTP/1.1" 200 - - - 12 11 "-" "aws-sdk-php/3.62.10 GuzzleHttp/6.3.3 curl/7.65.1 PHP/7.3.6" - JSzzQHXgaz+WnOmmIVhPk1D8+HHgU+lzZAmphAfRC7cF9KZGYy6z9NrAUgFGRGcpu6lUuG4dRxI= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com TLSv1.2 REST.GET.OBJECT urn%253Aoid%253A18610 "GET /urn%3Aoid%3A18610 HTTP/1.0" 200 - 452232 452232 39 37 "-" "aws-sdk-php/3.62.10" - eqmufJ/51zTwvlVoOCU2GgGjNdNfPuSUZyLinxCmikGaB9kTXuorc5cv/a2Zfj5I2KgfxLAnRsI= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com TLSv1.2 REST.HEAD.BUCKET - "HEAD / HTTP/1.1" 200 - - - 12 11 "-" "aws-sdk-php/3.62.10 GuzzleHttp/6.3.3 curl/7.65.1 PHP/7.3.6" - Aw0CLLgqqZ7QVQ+PripRGeKWJmztF6k04XgduCye2B7GEPmBt4y3bTc3jJrKKcIVmaWZTpFut5k= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com TLSv1.2 REST.GET.OBJECT urn%253Aoid%253A7376 "GET /urn%3Aoid%3A7376 HTTP/1.0" 200 - 148 148 70 70 "-" "aws-sdk-php/3.62.10" - CyKrBD2xZxEIKjl8LoamYH/OxyTXiBG3WYxAFPG4cqfjxBS3uzPkyG/ex/g+hLfEC9U29iz4BQo= SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader rxnc.s3.eu-central-1.amazonaws.com TLSv1.2

The log shows that ~50% of the requests made by nextcloud are HEAD /

The number of requests to s3 depends on Filesystem check frequency. Do you see less requests with "never"?

The number of requests to s3 depends on Filesystem check frequency. Do you see less requests with "never"?

Hello, this has happened with frequency checks set to 'never' or 0 as listed in the config.php. I have tested with and without filesystem checks and i have not seen any significant difference.
You have also linked a External Storages page but i use S3 as Primary Storage.

When i use nextcloud with S3 as primariy storage it is much slower than without S3 as primary storage.

I am using:
NC 16.0.3
Primary storage: S3 on Ceph

As i can see it takes:
login: 30-60 seconds
enter settings: 4 seconds
enter settings -> basic settings: 6 seconds
enter settings -> overview: 13 seconds
enter settings -> security: 10 seconds
enter apps: 6 seconds
back to files: 4 seconds

Could be better/faster ? :)

@gcommit

@gcommit How is this even related with this issue?

I was told there #7235 (comment)
To have a look at this issue.

@gcommit

@gcommit did you tried to follow @kesselb advice from message above?

@gcommit

@gcommit did you tried to follow @kesselb advice from message above?

i did.

Those using S3 should try to proxy their requests an implement a cache.
Guides (not Nexctloud specific, but you''l have an idea) for nginx:

https://gist.github.com/mikhailov/9639593
and
https://dpb587.me/post/2015/06/20/using-nginx-to-reverse-proxy-and-cache-s3-objects/

Also read comments of the author bellow the code (1st link)

#21728 could reduce the number of requests. Testers are always welcome ;)

@kesselb

#21728 could reduce the number of requests. Testers are always welcome ;)

This will be shipped in Nextcloud 20. So feedback on this is very welcome as @kesselb said.

Was it implemented in NC 20.0.1?

Navigation Menu

Repository navigation

S3: High connectivity check rate #16623

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration detail

Client configuration

Logs

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

S3: High connectivity check rate #16623

Description

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration detail

Client configuration

Logs

Activity

kesselb commented on Aug 5, 2019

casKd-dev commented on Aug 5, 2019

gcommit commented on Aug 9, 2019

casKd-dev commented on Aug 11, 2019

gcommit commented on Aug 11, 2019

RussianNeuroMancer commented on Oct 26, 2019

gcommit commented on Dec 16, 2019

solracsf commented on Jan 25, 2020

kesselb commented on Jul 7, 2020

MorrisJobke commented on Jul 15, 2020

IIPoliII commented on Nov 12, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions