Repository navigation
Unrecognized encoding #12861
Description
Steps to reproduce
$a=@'
ascii.txt
unicode.txt
utf32.txt
utf8.txt
utf8bom.txt
'@ -split "\s"
$url='https://raw.githubusercontent.com/he852100/111/master/Test/'
$a|%{"$((irm $url$_) -split '\s' -match '.')`t$_"}Expected behavior
123456你好 ascii.txt
123456你好 unicode.txt
123456你好 utf32.txt
123456你好 utf8.txt
123456你好 utf8bom.txt
Actual behavior
123456?? ascii.txt
��123456`O}Y unicode.txt
��123456`O}Y utf32.txt
123456你好 utf8.txt
(bom)123456你好 utf8bom.txt
Environment data
PSVersion 7.1.0-daily.20200530
PSEdition Core
GitCommitId 7.1.0-daily.20200530
OS Linux 4.9.112-perf #1 SMP PREEMPT Mon…
Platform Unix
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
Detecting charset=xxx has no purpose
Assuming it is a ps1, it will have serious compatibility issues
Activity
mklement0 commentedon Jun 1, 2020
To clarify the problem: the responses all indicate UTF-8 via the
charsetattribute in theContent-Typeheader field, but the actual payloads (file contents) are differently encoded, but indicate their specific encoding with a BOM.Currently, PowerShell only considers the
charset-specified encoding (in whose absence it currently assumes ISO-88591-), and ignores the BOM (even reading it as data).Arguably, a BOM in the payload, if present, should take precedence over the
charsetattribute.See #11547 (comment).