Why slugs returned in URI encoded format?

Hi there,
I have a post which it's slug is in Persian language, for example سلام دنیا. When I try to retrieve the post's slug, it's returned in unclear format like this:

%25d8%25b3%25d9%2584%25d8%25a7%25d9%2585-%25d8%25af%25d9%2586%25db%258c%25d8%25a7

While English slugs returned as is, And other fields like title and content that have Persian content, shows as well. Is it a true behavior? How can I get the true slug in Persian language?

oops, the reason is the slug was URI encoded and I have to use decodeURI to make the slug string as what I expect.
So the question is why slug should be returned in URI encoded format while the slugs saved in URL friendly format in backend?

Good catch. This is how it's stored in the database, which is why we're returning it that way. The dashboard is actually decoding this when it displays it to you, which is why you see it as the original text.

I'm not sure whether we should decode this before returning it, since it is stored in the DB this way, and there could conceivably be cases where you lose data because of hidden encoding/decoding.

@WP-API/amigos Thoughts here?

@nylen

This is how it's stored in the database, which is why we're returning it that way.

Why is it stored in the database this way?

I'm not sure whether we should decode this before returning it, since it is stored in the DB this way, and there could conceivably be cases where you lose data because of hidden encoding/decoding.

I'm not familiar enough with character encoding to be able to make a determination one way or the other.

@nylen How do you handle this on WPcom? Do you have any prior art or conversations you can pull up?

Related #1227 (comment)

IMO if it's stored in the DB this way, we should return that in the API. If web client also shows that link as the dashboard does, it will similarly be decoded.

For reference, get_terms() permits querying by either representation:

salty-wordpress ➜  wordpress-develop.dev  wp term list post_tag
+---------+------------------+----------+-----------------------------------+-------------+--------+-------+
| term_id | term_taxonomy_id | name     | slug                              | description | parent | count |
+---------+------------------+----------+-----------------------------------+-------------+--------+-------+
| 3       | 3                | Foo      | foo                               |             | 0      | 0     |
| 2       | 2                | سلام دنی | %d8%b3%d9%84%d8%a7%d9%85-%d8%af%d9%86%db%8c |             | 0      | 0     |
+---------+------------------+----------+-----------------------------------+-------------+--------+-------+
salty-wordpress ➜  wordpress-develop.dev  wp shell
wp> get_terms( 'post_tag', array( 'slug' => 'foo', 'hide_empty' => 0 ) );
array(1) {
  [0]=>
  object(WP_Term)#767 (10) {
    ["term_id"]=>
    int(3)
    ["name"]=>
    string(3) "Foo"
    ["slug"]=>
    string(3) "foo"
    ["term_group"]=>
    int(0)
    ["term_taxonomy_id"]=>
    int(3)
    ["taxonomy"]=>
    string(8) "post_tag"
    ["description"]=>
    string(0) ""
    ["parent"]=>
    int(0)
    ["count"]=>
    int(0)
    ["filter"]=>
    string(3) "raw"
  }
}
wp> get_terms( 'post_tag', array( 'slug' => '%d8%b3%d9%84%d8%a7%d9%85-%d8%af%d9%86%db%8c', 'hide_empty' => 0 ) );
array(1) {
  [0]=>
  object(WP_Term)#824 (10) {
    ["term_id"]=>
    int(2)
    ["name"]=>
    string(15) "سلام دنی"
    ["slug"]=>
    string(43) "%d8%b3%d9%84%d8%a7%d9%85-%d8%af%d9%86%db%8c"
    ["term_group"]=>
    int(0)
    ["term_taxonomy_id"]=>
    int(2)
    ["taxonomy"]=>
    string(8) "post_tag"
    ["description"]=>
    string(0) ""
    ["parent"]=>
    int(0)
    ["count"]=>
    int(0)
    ["filter"]=>
    string(3) "raw"
  }
}
wp> get_terms( 'post_tag', array( 'slug' => 'سلام دنی', 'hide_empty' => 0 ) );
array(1) {
  [0]=>
  object(WP_Term)#826 (10) {
    ["term_id"]=>
    int(2)
    ["name"]=>
    string(15) "سلام دنی"
    ["slug"]=>
    string(43) "%d8%b3%d9%84%d8%a7%d9%85-%d8%af%d9%86%db%8c"
    ["term_group"]=>
    int(0)
    ["term_taxonomy_id"]=>
    int(2)
    ["taxonomy"]=>
    string(8) "post_tag"
    ["description"]=>
    string(0) ""
    ["parent"]=>
    int(0)
    ["count"]=>
    int(0)
    ["filter"]=>
    string(3) "raw"
  }
}

fantastic.

@joehoyle

@joehoyle I'm using wp-api with my angular app and the links created with the encoded slug, will not decoded in browser automatically, else I decode theme manually as this way:

$rootScope.decodeFa = function(str) {
    return decodeURI(str);
}

So there is two ways:

Release the encoded slugs AS IS and notice users this problem to let theme make their own approaches to decode or not or something else.
Return the encoded slug.

Why is it stored in the database this way?

So, I've been thinking about this, and I think it actually makes sense. The slug is intended as the slug (duh) for the URL, and the primary (and basically only) use case is in URLs. It makes sense then that this value needs to be safe for embedding in URLs, and that invalid characters are encoded.

I'd say this is a wontfix, but we should document this in the schema as being specifically URL-safe data, and that invalid characters need to be encoded.

My understanding is that storing content encoded in the DB was done before the best practice of encoding immediately before display was firmly established and recognized. From https://codex.wordpress.org/Data_Validation:

Tip: It's best to do the output validation as late as possible, ideally as it's being outputted, as opposed to further up in your script. This way you can always be sure that your data is properly validated/escaped and you don't need to remember if the variable has been previously validated.

Having to deal with already-encoded data is more tricky because you are forced to violate this rule, and there can be bugs associated with double-encoding.

I couldn't find any discussions on slugs in particular on the WPCOM side, probably because as you've mentioned they are only used in a URL context which isn't so bad to deal with. My experience with these issues has been more related to HTML entities (#1227 (comment)).

IMO it would be a win for application developers to have the API decode this data. But a pretty minor issue in the scheme of things.

I disagree that this is necessarily a legacy thing, I think it's intentional. If the field was a URL value, we'd expect the URL encoded data, as we're saying "this is a valid URL that we are returning". The value here isn't a full URL, but it is URL data, so it makes sense to only return valid URL characters.

For example, if I had a slug like foo\bar, I'd expect a URL field to have http://example.com/foo%5Cbar, so it makes sense for the slug field to have foo%5Cbar. The only time I think it may be confusing is when saving data.

No further discussion has happened here in a while, so going to close this out.

What will happen if we send the non-escaped version to WP in a request, e.g. we PUT the unescaped version when we got the escaped version out? Will WP re-escape when it inserts into the DB?

Update from my question: Yes, if you PUT the Arabic text as the slug, it will be inserted into the DB as escaped, while it will be untransformed in the content, excerpt and title field. This matches the URL-oriented purpose of the slugs.

Navigation Menu

Repository navigation

Why slugs returned in URI encoded format? #2243

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Why slugs returned in URI encoded format? #2243

Description

Activity

bobsilon commented on Feb 10, 2016

rmccue commented on Feb 10, 2016

danielbachhuber commented on Feb 10, 2016

joehoyle commented on Feb 10, 2016

danielbachhuber commented on Feb 10, 2016

joehoyle commented on Feb 10, 2016

bobsilon commented on Feb 10, 2016

rmccue commented on Feb 11, 2016

nylen commented on Feb 14, 2016

rmccue commented on Feb 15, 2016

rmccue commented on Sep 12, 2016

kadamwhite commented on Sep 13, 2016

kadamwhite commented on Sep 22, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions