Repository navigation
Webcmdlets should parse the <html><head><meta charset="foo"> attribute for the correct encoding if not in http header #3267
Description
Activity
I believe we should discuss here a default encoding for Web cmdlets.
Currently we use RFC 2616 "ISO-8859-1"
In fact, browsers for a long time use Windows-1252.
And HTML5 default is UTF-8.Perhaps we should also use Windows 1252.
Reacted by Michael KlementI looked Web cmdlets and found that we use
ContentTypeparameter to encode a request (Content-Type header can containscharsetvalue). If we don't specify a charset inContenTypewe use default charset ISO-8859-1.
For decoding a response we use a charset fromContentTypeof the response. If a server return noContentTypewe use the same default charset ISO-8859-1.So we should treat
-CharSetas-ResponseCharSet.After some thought, I believe that using Windows 1252 as default is obsolete and we should aim at HTML5 and UTF-8 as defaults. https://w3techs.com/technologies/details/ml-html5/all/all
Reacted by Michael Klement and joshuatee@PowerShell/powershell-committee reviewed this and agree this is an issue for customers. proposal is to parse the HTTP header first, if charset is in content-type, we use that. otherwise if content-type is html, we parse
<meta charset="X">for the charset attribute and use thatReacted by Ilya, Michael Klement and joshuatee- changed the title
[-]Expose -CharSet parameter to webcmdlets[/-][+]Webcmdlets should parse the <html><head><meta charset="foo"> attribute for the correct encoding if not in http header[/+]on Mar 9, 2017 In Windows Powershell we use Internet Explorer to parse HTML. What portable parser we can use in PowerShell Core?
And if HTML don't contain
<meta charset="foo">? What defaults we should use for fallback?Ilya (@iSazonov) the proposal is that we don't rely on any browser for the html parsing (if complete parsing is needed, I still think it would make more sense in a
convertfrom-htmlcmdlet). I think "best effort" for this is sufficient (perhaps even just a regular expression) to cover the majority of cases and we wouldn't worry about malformed html.Steve Lee (@SteveL-MSFT) Original Windows web cmdlet returns
ParsedHtml : mshtml.HTMLDocumentClass- we want lost the functionality?Ilya (@iSazonov)
.ParshedHtmlrelied on Internet Explorer. I don't think we can have a dependency on any particular web browser in the webcmdlets.To answer your other question I missed, if
<meta charset>doesn't exist andcharsetisn't specified in thecontent-typeHTTP header, then we do what we do today which is assume ISO-8859-1.ParshedHtml relied on Internet Explorer
If we use any ported library for HTML parsing we will solve this Issue, get
ParshedHtmlported, as well as get a base forConvertFrom-HTML.I wonder about
ISO-8859-1. Why we do not want to accept the new standard HTML5?48 remaining items
- added this to the milestone
on Nov 3, 2017 Any further work on this I'm deferring to 6.1.0
Submitted RFC for the creation of
ConvertFrom-Htmlhere: PowerShell/PowerShell-RFC#137Reacted by Ilya and Brendan McCoyWe get new HttpClient with .Net 3+ so I remove the label.
Some websites do not populate the
charsetproperty of thecontent-typeheader so characters aren't rendered correctly. Suggestion is to expose a-charsetparameter, however the user still needs to know the expected charset. Advanced users today can do the encoding translation in script.utf-8probably works in most cases, so not entirely sure how useful this will be to expect the user to know ahead of time the correct charset.See discussion from #3126 for details on how this came about