Repository navigation
Webcmdlets should parse the <html><head><meta charset="foo"> attribute for the correct encoding if not in http header #3267
Description
Some websites do not populate the charset property of the content-type header so characters aren't rendered correctly. Suggestion is to expose a -charset parameter, however the user still needs to know the expected charset. Advanced users today can do the encoding translation in script. utf-8 probably works in most cases, so not entirely sure how useful this will be to expect the user to know ahead of time the correct charset.
See discussion from #3126 for details on how this came about
Activity
iSazonov commentedon Mar 6, 2017
I believe we should discuss here a default encoding for Web cmdlets.
Currently we use RFC 2616 "ISO-8859-1"
In fact, browsers for a long time use Windows-1252.
And HTML5 default is UTF-8.
Perhaps we should also use Windows 1252.
iSazonov commentedon Mar 7, 2017
I looked Web cmdlets and found that we use
ContentTypeparameter to encode a request (Content-Type header can containscharsetvalue). If we don't specify a charset inContenTypewe use default charset ISO-8859-1.For decoding a response we use a charset from
ContentTypeof the response. If a server return noContentTypewe use the same default charset ISO-8859-1.So we should treat
-CharSetas-ResponseCharSet.After some thought, I believe that using Windows 1252 as default is obsolete and we should aim at HTML5 and UTF-8 as defaults. https://w3techs.com/technologies/details/ml-html5/all/all
SteveL-MSFT commentedon Mar 9, 2017
@PowerShell/powershell-committee reviewed this and agree this is an issue for customers. proposal is to parse the HTTP header first, if charset is in content-type, we use that. otherwise if content-type is html, we parse
<meta charset="X">for the charset attribute and use that[-]Expose -CharSet parameter to webcmdlets[/-][+]Webcmdlets should parse the <html><head><meta charset="foo"> attribute for the correct encoding if not in http header[/+]iSazonov commentedon Mar 9, 2017
In Windows Powershell we use Internet Explorer to parse HTML. What portable parser we can use in PowerShell Core?
And if HTML don't contain
<meta charset="foo">? What defaults we should use for fallback?SteveL-MSFT commentedon Mar 9, 2017
Ilya (@iSazonov) the proposal is that we don't rely on any browser for the html parsing (if complete parsing is needed, I still think it would make more sense in a
convertfrom-htmlcmdlet). I think "best effort" for this is sufficient (perhaps even just a regular expression) to cover the majority of cases and we wouldn't worry about malformed html.iSazonov commentedon Mar 9, 2017
Steve Lee (@SteveL-MSFT) Original Windows web cmdlet returns
ParsedHtml : mshtml.HTMLDocumentClass- we want lost the functionality?SteveL-MSFT commentedon Mar 9, 2017
Ilya (@iSazonov)
.ParshedHtmlrelied on Internet Explorer. I don't think we can have a dependency on any particular web browser in the webcmdlets.To answer your other question I missed, if
<meta charset>doesn't exist andcharsetisn't specified in thecontent-typeHTTP header, then we do what we do today which is assume ISO-8859-1iSazonov commentedon Mar 9, 2017
Steve Lee (@SteveL-MSFT)
If we use any ported library for HTML parsing we will solve this Issue, get
ParshedHtmlported, as well as get a base forConvertFrom-HTML.I wonder about
ISO-8859-1. Why we do not want to accept the new standard HTML5?48 remaining items
SteveL-MSFT commentedon Nov 3, 2017
Any further work on this I'm deferring to 6.1.0
MSAdministrator commentedon Jul 15, 2018
Submitted RFC for the creation of
ConvertFrom-Htmlhere: PowerShell/PowerShell-RFC#137iSazonov commentedon Jul 16, 2020
We get new HttpClient with .Net 3+ so I remove the label.