MySQL 4byte support by nickvergessen · Pull Request #1773 · nextcloud/server

nickvergessen · 2016-10-18T07:49:26Z

Requires Reduce the number of 4000chars columns activity#53

@MorrisJobke can you setup drone with 4byte?

mention-bot · 2016-10-18T07:49:28Z

@nickvergessen, thanks for your PR! By analyzing the history of the files in this pull request, we identified @icewind1991, @stweil and @DeepDiver1975 to be potential reviewers.

MorrisJobke · 2016-10-18T07:50:39Z

@MorrisJobke can you setup drone with 4byte?

Yes.

MorrisJobke · 2016-10-18T08:20:08Z

-      PHP: 7.1
-    - DB: mysql
-      PHP: 5.6
-    - DB: postgres


Because I currently can't test locally I removed this to have faster feedback ;)

MorrisJobke · 2016-10-18T08:35:21Z

I rebased to squash the testing commits

MorrisJobke · 2016-10-18T08:36:24Z

@nickvergessen Your turn now :)

nickvergessen · 2016-10-18T09:11:42Z

Current problem:
Update with activity app fails. We try to change the collation Repair info: Change collation for oc_activity before the database is updated.
This results in:

Syntax error or access violation: 1118 Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs

Which is what would be solved by nextcloud/activity#53
But the repair step is executed before the update. My suggestion would be to ignore this error in the before step and retry it after the update was completed.

I commited that

MorrisJobke · 2016-10-18T12:59:17Z

1) Test\Repair\RepairCollationTest::testCollationConvert
Argument 2 passed to OC\Repair\Collation::__construct() must implement interface OCP\ILogger, instance of OC\DB\Connection given, called in /drone/src/github.com/nextcloud/server/tests/lib/Repair/RepairCollationTest.php on line 66 and defined
/drone/src/github.com/nextcloud/server/lib/private/Repair/Collation.php:54
/drone/src/github.com/nextcloud/server/tests/lib/Repair/RepairCollationTest.php:66

MorrisJobke · 2016-10-18T14:19:16Z

3) Test\Files\Cache\CacheTest::testFolder with data set #2 ('😐, NEUTRAL_FACE, UTF-8 hex F09F9890')
MySQL doesn't support 4 byte UTF-8
/drone/src/github.com/nextcloud/server/tests/lib/Files/Cache/CacheTest.php:117
4) Test\Files\Cache\MoveFromCacheTraitTest::testFolder with data set #2 ('😐, NEUTRAL_FACE, UTF-8 hex F09F9890')
MySQL doesn't support 4 byte UTF-8
/drone/src/github.com/nextcloud/server/tests/lib/Files/Cache/CacheTest.php:117
5) Test\Files\Cache\Wrapper\CacheJailTest::testFolder with data set #2 ('😐, NEUTRAL_FACE, UTF-8 hex F09F9890')
MySQL doesn't support 4 byte UTF-8
/drone/src/github.com/nextcloud/server/tests/lib/Files/Cache/CacheTest.php:117
6) Test\Files\Cache\Wrapper\CachePermissionsMaskTest::testFolder with data set #2 ('😐, NEUTRAL_FACE, UTF-8 hex F09F9890')
MySQL doesn't support 4 byte UTF-8
/drone/src/github.com/nextcloud/server/tests/lib/Files/Cache/CacheTest.php:117

codecov-io · 2016-10-18T15:24:21Z

Current coverage is 57.24% (diff: 17.39%)

Merging #1773 into master will decrease coverage by 0.02%

@@             master      #1773   diff @@
==========================================
  Files          1075       1075          
  Lines         61278      61314    +36   
  Methods        6864       6865     +1   
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          35090      35097     +7   
- Misses        26188      26217    +29   
  Partials          0          0

Diff Coverage	File Path
0%	core/register_command.php
0%	lib/private/Repair/Collation.php
0%	lib/private/Repair.php
0%	lib/private/Setup/MySQL.php
0%	lib/private/Setup/AbstractDatabase.php
0%	config/config.sample.php
0%	lib/private/DB/AdapterMySQL.php
0%	...Builder/ExpressionBuilder/MySqlExpressionBuilder.php
•••• 40%	lib/private/DB/ConnectionFactory.php
••••• 50%	lib/private/DB/MDB2SchemaWriter.php

Review all 12 files changed

Powered by Codecov. Last update 129fdf2...10049d0

* success on SQLite and Postgres * failure on MySQL due to the limited charset that only supports up to 3 bytes

* fully optional * requires additional options set in the database

Signed-off-by: Joas Schilling <[email protected]>

Signed-off-by: Morris Jobke <[email protected]>

Signed-off-by: Joas Schilling <[email protected]>

Signed-off-by: Morris Jobke <[email protected]>

Signed-off-by: Joas Schilling <[email protected]>

MorrisJobke · 2016-10-18T22:15:40Z

Rebased for CI

nickvergessen · 2016-10-19T07:47:17Z

Test pass, ready to go.

Second review @rullzer @LukasReschke

MorrisJobke · 2016-10-19T07:50:32Z

👍

LukasReschke · 2016-10-19T20:09:06Z

LGTM

nickvergessen added enhancement 2. developing Work in progress labels Oct 18, 2016

nickvergessen added this to the Nextcloud 11.0 milestone Oct 18, 2016

MorrisJobke self-assigned this Oct 18, 2016

MorrisJobke reviewed Oct 18, 2016

View reviewed changes

MorrisJobke force-pushed the testing-characters-in-db branch 2 times, most recently from 290cebd to bc4f76c Compare October 18, 2016 08:35

MorrisJobke removed their assignment Oct 18, 2016

nickvergessen added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Oct 18, 2016

MorrisJobke and others added 12 commits October 19, 2016 00:15

Adding tests for 4 byte unicode characters

972e560

* success on SQLite and Postgres * failure on MySQL due to the limited charset that only supports up to 3 bytes

Add config option to update charset of mysql to utf8mb4

cc28f82

* fully optional * requires additional options set in the database

only disable unicode test on mysql

296a327

Fixing ctor call

a7245ea

Adding docker based unit test execution for mysql utf8mb4

9c3f066

fix docker

d294ef2

Correctly save and pass on the charset

9356a0e

Signed-off-by: Joas Schilling <[email protected]>

add 4 byte mysql test run

d0a3d17

Signed-off-by: Morris Jobke <[email protected]>

Ignore failures of collation change in the pre update step

15bbe02

Signed-off-by: Joas Schilling <[email protected]>

Fix the test

17a2723

Signed-off-by: Joas Schilling <[email protected]>

Fix like queries in the QueryBuilder

64c9ef9

Signed-off-by: Joas Schilling <[email protected]>

test alternative drone syntax for command options

b1235a6

Signed-off-by: Morris Jobke <[email protected]>

nickvergessen added 2 commits October 19, 2016 00:15

Fix test of repair step

43b7b14

Signed-off-by: Joas Schilling <[email protected]>

Do not skip when mysql uses utf8mb4

303e073

Signed-off-by: Joas Schilling <[email protected]>

MorrisJobke force-pushed the testing-characters-in-db branch from e5bfad1 to 303e073 Compare October 18, 2016 22:15

LukasReschke merged commit 73f4ae9 into master Oct 19, 2016

LukasReschke deleted the testing-characters-in-db branch October 19, 2016 20:09

nickvergessen mentioned this pull request Oct 24, 2016

4 byte unicode characters in filenames with MariaDB causes severe bugs #1560

Closed

BernhardPosselt mentioned this pull request Nov 10, 2016

Mysql error nextcloud/news#53

Closed

FriedCircuits mentioned this pull request Jan 18, 2017

Enable utf8mb4 nextcloud/vm#97

Closed

cweiske mentioned this pull request Sep 23, 2018

Initial sync with Tomdroid not working / emoji utf8 problem grosjo/nextcloud-grauphel#64

Closed

Navigation Menu

Repository navigation

MySQL 4byte support - #1773

MySQL 4byte support#1773
LukasReschke merged 14 commits into
masterfrom
testing-characters-in-db

nickvergessen commented Oct 18, 2016 •

edited by LukasReschke

Loading

Uh oh!

mention-bot commented Oct 18, 2016

Uh oh!

MorrisJobke commented Oct 18, 2016

Uh oh!

MorrisJobke Oct 18, 2016

Uh oh!

MorrisJobke commented Oct 18, 2016

Uh oh!

MorrisJobke commented Oct 18, 2016

Uh oh!

nickvergessen commented Oct 18, 2016

Uh oh!

MorrisJobke commented Oct 18, 2016

Uh oh!

MorrisJobke commented Oct 18, 2016

Uh oh!

codecov-io commented Oct 18, 2016 •

edited

Loading

Uh oh!

MorrisJobke commented Oct 18, 2016

Uh oh!

nickvergessen commented Oct 19, 2016

Uh oh!

MorrisJobke commented Oct 19, 2016

Uh oh!

LukasReschke commented Oct 19, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

nickvergessen commented Oct 18, 2016 • edited by LukasReschke Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mention-bot commented Oct 18, 2016

Uh oh!

MorrisJobke commented Oct 18, 2016

Uh oh!

MorrisJobke Oct 18, 2016

Choose a reason for hiding this comment

Uh oh!

MorrisJobke commented Oct 18, 2016

Uh oh!

MorrisJobke commented Oct 18, 2016

Uh oh!

nickvergessen commented Oct 18, 2016

Uh oh!

MorrisJobke commented Oct 18, 2016

Uh oh!

MorrisJobke commented Oct 18, 2016

Uh oh!

codecov-io commented Oct 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current coverage is 57.24% (diff: 17.39%)

Uh oh!

MorrisJobke commented Oct 18, 2016

Uh oh!

nickvergessen commented Oct 19, 2016

Uh oh!

MorrisJobke commented Oct 19, 2016

Uh oh!

LukasReschke commented Oct 19, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nickvergessen commented Oct 18, 2016 •

edited by LukasReschke

Loading

codecov-io commented Oct 18, 2016 •

edited

Loading