URL: percent-encoding test framework basics by annevk · Pull Request #26317 · web-platform-tests/wpt

annevk · 2020-10-28T15:13:19Z

@domenic @rmisev @achristensen07 thoughts? (There are many ways to expand this to also cover form submission and such, but I thought I'd start out simple to ensure we agree on a common ground.)

domenic

Excited to see further work on data-driven tests!!

domenic · 2020-10-28T15:52:31Z

+    value = request.GET.first(b"value")
+    encoding = request.GET.first(b"encoding")
+
+    output_value = numeric_references(unquote(value).decode(b"utf-8"))


I worry a bit about all the steps of smuggling the value through URL parameters. If a browser doesn't do percent-encoding correctly, then probably it will fail to get the correct output_value, messing up the test.

I guess the assumption is that the browser does utf-8 percent encoding correctly?

If you wanted to avoid that assumption, you could use another encoding scheme, e.g. base64 encoding or maybe JSON.stringify.

Yeah, that might be a good improvement here.

It doesn't seem like the output of JSON.stringify is always ASCII so maybe I'll leave this as-is for now.

base64 with atob/btoa and Python's equivalent seems like it'd work nicely?

How does that work given the non-ASCII input?

Ah, right, it's indeed a bit more complicated. https://github.com/jsdom/whatwg-url/blob/d5f65c2bef5d88f873203d33289ddfe13f2eee1f/live-viewer/live-viewer.js#L135-L150 is what I was thinking, but perhaps it's not worth it, especially given that you'd have to implement it in both Python and JS.

I made this work, but it took longer than expected due to some weirdness from WPT.

annevk · 2020-10-30T11:22:00Z

I filed https://bugs.webkit.org/show_bug.cgi?id=218380 and https://bugzilla.mozilla.org/show_bug.cgi?id=1674369 while playing around a bit more.

domenic

Interesting that every browser seems to fail some of these... but different ones.

domenic · 2020-10-30T15:29:19Z

+    value = request.GET.first(b"value")
+    encoding = request.GET.first(b"encoding")
+
+    output_value = numeric_references(unquote(value).decode(b"utf-8"))


base64 with atob/btoa and Python's equivalent seems like it'd work nicely?

Since the ISO-2022-JP encoder is stateful, percent-encoding needs to hold onto an instance of the encoder and manually perform error handling. This also requires the input to be the full string rather than individual code points as otherwise the callers of percent-encoding would need to be aware of this too. (As UTF-8 encoding cannot fail this problem does not affect those endpoints.) Builds on this Encoding PR: whatwg/encoding#238. Tests: web-platform-tests/wpt#26158 and web-platform-tests/wpt#26317. Fixes #557.

andreubotella · 2020-11-03T11:16:43Z

This test case should probably be added as well, since the bug that makes Firefox fail on "−" with Shift_JIS is more general than that:

{
  "input": "á|",
  "output": {
    "windows-1252": "%E1|",
    "utf-8": "%C3%A1|"
  }
}

domenic · 2020-11-03T15:01:09Z

+      "windows-1252": "%86"
+    }
+  },
+  "This uses a trailing A to prevent the URL parser from trimming the C0 control.",


It looks like in to-ascii.json we had a "comment" field inside each test case, which we could use for single-test case comments. Might be worth reusing here. (whatwg-url creates the test case description by appending the comment, if it exists.)

Follows web-platform-tests/wpt#26317.

domenic · 2020-11-03T15:07:02Z

Still LGTM. I incorporated these into whatwg-url, although as discussed previously whatwg-url doesn't do non-UTF-8 encodings, so it's not as helpful as one might like. Still, it gives a basic sanity check, I guess. jsdom/whatwg-url#174

Refs: web-platform-tests/wpt#26317

Refs: web-platform-tests/wpt#26317 PR-URL: #36032 Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Joyee Cheung <[email protected]>

Follows web-platform-tests/wpt#26317.

wpt-pr-bot added the url label Oct 28, 2020

wpt-pr-bot assigned mikewest Oct 28, 2020

wpt-pr-bot requested review from GPHemsley, TimothyGu, domenic and mikewest October 28, 2020 15:13

annevk mentioned this pull request Oct 28, 2020

Refactor query state to operate on a buffer whatwg/url#558

Merged

3 tasks

wpt-pr-bot temporarily deployed to wpt-preview-26317 October 28, 2020 15:49 Inactive

domenic reviewed Oct 28, 2020

View reviewed changes

annevk requested a review from domenic October 30, 2020 11:30

wpt-pr-bot temporarily deployed to wpt-preview-26317 October 30, 2020 11:32 Inactive

domenic approved these changes Oct 30, 2020

View reviewed changes

wpt-pr-bot temporarily deployed to wpt-preview-26317 November 3, 2020 10:56 Inactive

wpt-pr-bot temporarily deployed to wpt-preview-26317 November 3, 2020 11:20 Inactive

domenic reviewed Nov 3, 2020

View reviewed changes

domenic added a commit to jsdom/whatwg-url that referenced this pull request Nov 3, 2020

Incorporate the latest web platform percent-encoding tests

43f2d46

Follows web-platform-tests/wpt#26317.

domenic mentioned this pull request Nov 3, 2020

Incorporate the latest web platform percent-encoding tests jsdom/whatwg-url#174

Merged

wpt-pr-bot temporarily deployed to wpt-preview-26317 November 3, 2020 15:20 Inactive

annevk added 6 commits November 3, 2020 16:50

URL: percent-encoding test framework basics

6d5d7f8

add some more tests and documentation

b771496

use base64

6d96ea7

oops

3e4cade

add another test for a Fx failure

dfaa767

oops again

1404791

annevk force-pushed the annevk/pe branch from 229cee7 to 1404791 Compare November 3, 2020 15:51

wpt-pr-bot deployed to wpt-preview-26317 November 3, 2020 15:55 View deployment

annevk merged commit 09d8830 into master Nov 3, 2020

annevk deleted the annevk/pe branch November 3, 2020 16:00

watilde added a commit to watilde/node that referenced this pull request Nov 8, 2020

test: update wpt url and resource

c978d9f

Refs: web-platform-tests/wpt#26317

watilde mentioned this pull request Nov 8, 2020

test: update wpt url and resource nodejs/node#36032

Closed

3 tasks

domenic added a commit to jsdom/whatwg-url that referenced this pull request Feb 23, 2021

Incorporate the latest web platform percent-encoding tests

68fd22c

Follows web-platform-tests/wpt#26317.

domenic added a commit to jsdom/whatwg-url that referenced this pull request Feb 23, 2021

Incorporate the latest web platform percent-encoding tests

ab15745

Follows web-platform-tests/wpt#26317.

domenic added a commit to jsdom/whatwg-url that referenced this pull request Feb 23, 2021

Incorporate the latest web platform percent-encoding tests

ff395c3

Follows web-platform-tests/wpt#26317.

domenic added a commit to jsdom/whatwg-url that referenced this pull request Feb 23, 2021

Incorporate the latest web platform percent-encoding tests

385d54d

Follows web-platform-tests/wpt#26317.

Uh oh!

Conversation

annevk commented Oct 28, 2020

Uh oh!

domenic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

annevk commented Oct 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

domenic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andreubotella commented Nov 3, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

domenic commented Nov 3, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

annevk commented Oct 30, 2020 •

edited

Loading