cron.php OCA\DAV\CardDAV\SyncJob causes a high load daily

Steps to reproduce

Have a lot of users. E.g. 5000
Wait for the OCA\DAV\CardDAV\SyncJob cron job to run (happens once in 24h)

Expected behaviour

Job should run quickly and the load on the server should not go higher.

Actual behaviour

It runs for hours depending on the amount of users. For approximately 5000 users it needs 3 hours. It tries to parse all users and their addressbook cards in one pass, which causes a high load on the database and takes up more server resource than it should.

Server configuration

Operating system: Debian 9.1 (stretch)

Web server: Nginx 1.10.3

Database: MySQL 5.8

PHP version: 7

Nextcloud version: 12.0.2

Contacts version: 2.0.1

Updated from an older Nextcloud or fresh install: Updated from Nextcloud version 12

Signing status:

No errors have been found.

List of activated apps:

Enabled:
  - activity: 2.5.2
  - admin_audit: 1.2.0
  - calendar: 1.5.6
  - comments: 1.2.0
  - contacts: 2.0.1
  - dav: 1.3.0
  - encryption: 1.6.0
  - federatedfilesharing: 1.2.0
  - federation: 1.2.0
  - files: 1.7.2
  - files_pdfviewer: 1.1.1
  - files_sharing: 1.4.0
  - files_texteditor: 2.4.1
  - files_trashbin: 1.2.0
  - files_videoplayer: 1.1.0
  - gallery: 17.0.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.0.0
  - nextcloud_announcements: 1.1
  - notes: 2.3.1
  - notifications: 2.0.0
  - oauth2: 1.0.5
  - password_policy: 1.2.2
  - provisioning_api: 1.2.0
  - serverinfo: 1.2.0
  - sharebymail: 1.2.0
  - survey_client: 1.0.0
  - systemtags: 1.2.0
  - tasks: 0.9.5
  - twofactor_backupcodes: 1.1.1
  - updatenotification: 1.2.0
  - workflowengine: 1.2.0

Nextcloud configuration:

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "12.0.2.0",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "forcessl": true,
        "theme": "***REMOVED SENSITIVE VALUE***",
        "enable_previews": true,
        "enable_avatars": false,
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "log_authfailip": true,
        "loglevel": 2,
        "maintenance": false,
        "customclient_desktop": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "share_folder": "\/Shared",
        "overwritewebroot": "\/",
        "overwriteprotocol": "https",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "singleuser": false,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "trashbin_retention_obligation": "auto,30",
        "activity_expire_days": 14,
        "logtimezone": "Europe\/Zurich",
        "log_type": "errorlog",
        "skeletondirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***"
    }
}

Are you using encryption: yes

Client configuration

Browser:
not relevant because it is a CLI/server side issue
Operating system:
not relevant because it is a CLI/server side issue

Logs

Web server error log

no errors

Nextcloud log (data/nextcloud.log)

Nextcloud log

no errors

Browser log

no errors

@georgehrke

cc @georgehrke

@CamZie

@CamZie Do you have a lot of federated shares?

@georgehrke

@georgehrke we do not have any federated shares.

I would like to ask if there are any news for this problem?

I would like to ask if there are any news for this problem?

Not yet. I forgot to ask: Do you use any user-backend? If yes, how fast is the connection to the user-backend?

I'm asking because OCA\DAV\CardDAV\SyncJob does not touch any user address book. All it does is creating/updating VCards for each user in the system address book and if you use federated shares sync the system address book with other instances.

Thank you for the information. We are not using any user-backend and simply have all the users in the MySQL database.

Is it possible that the cause of this problem is the OCA\DAV\CardDAV\SyncJob, which creates/updates the vCards for each users in one try?

Is it possible that the cause of this problem is the OCA\DAV\CardDAV\SyncJob, which creates/updates the vCards for each users in one try?

So you are not sure that the high load was caused by OCA\DAV\CardDAV\SyncJob, but it was just a wild guess?

Can you check in the oc_jobs table what other jobs are scheduled for the instance? Maybe the last_run column will give a hint what job took so much time.

We were sure that this was caused by the OCA\DAV\CardDAV\SyncJob, because we also saw the following entry on the database:

id: 565626
class: OCA\DAV\CardDAV\SyncJob
argument: null
last_run: 1512512102
last_checked: 1512549901
reserved_at: 0
execution_duration: 9260

@rullzer

Looks like we need to optimise that job then. @rullzer @schiessle @blizzz Any idea for this?

I'd have to run test. But I see why it fails.

The first part of the job is actually playing nice I'd say: https://github.com/nextcloud/server/blob/master/apps/dav/lib/CardDAV/SyncService.php#L320-L325

Just for each user update that user which seems fine (and makes sense).

However the second part: https://github.com/nextcloud/server/blob/master/apps/dav/lib/CardDAV/SyncService.php#L328

Creates a huge structure in RAM. We should probably try to add a generator or something here... I'll see if I can add 10k users easily to my dev setup and see it perform

Mmmm so thinking more about this... why don't we just listen to the postDelete hook properly... and then drop the user from the sytemaddressbook...

Aaaah because the mapping is reversed... (sorry for the live debug log :P)

Ok so I was wrong. https://github.com/nextcloud/server/blob/master/apps/dav/lib/CardDAV/SyncService.php#L328 is actually very fast.

https://github.com/nextcloud/server/blob/master/apps/dav/lib/CardDAV/SyncService.php#L321 is slow

This is because we regenerate the vcard all the time. Which requires fetching an avatar. Which goes to disk etc.

However the more I look at it the more I wonder why we have this job. This should all be handled by the hooks in place already. And for the times we don't let users just use the occ command I'd say.

So I just checked.

Creating a user => Creates it in the systemaddressbook
Deleting a user => Deletes it from the systemaddressbook
Updating a user => Updates it in the systemaddressbook

So I think we can just remove the cronjob for this.
All user backends should properly emit those hooks and if not we should fix them.

Worst comes to worst users can always trigger a sync via the occ command.

Do no run SyncJob in cron

PR in #7644

Do no run SyncJob in cron

Navigation Menu

Repository navigation

cron.php OCA\DAV\CardDAV\SyncJob causes a high load daily #7184

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

Client configuration

Logs

Web server error log

Nextcloud log (data/nextcloud.log)

Browser log

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

cron.php OCA\DAV\CardDAV\SyncJob causes a high load daily #7184

Description

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

Client configuration

Logs

Web server error log

Nextcloud log (data/nextcloud.log)

Browser log

Activity

MorrisJobke commented on Nov 15, 2017

georgehrke commented on Nov 15, 2017

CamZie commented on Nov 15, 2017

CamZie commented on Nov 29, 2017

georgehrke commented on Nov 29, 2017

CamZie commented on Nov 29, 2017

georgehrke commented on Dec 4, 2017

CamZie commented on Dec 6, 2017

MorrisJobke commented on Dec 20, 2017

rullzer commented on Dec 21, 2017

rullzer commented on Dec 21, 2017

rullzer commented on Dec 21, 2017

rullzer commented on Dec 21, 2017

rullzer commented on Dec 21, 2017

rullzer commented on Dec 29, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions