Resolve UTF-8, UTF-16, ASCII inconsistencies

PS> ls > test
BASH> ls >> test
BASH> cat test # Content looks correct
PS> cat test # Content is wrong at the end

Need to discuss this at greater length. Some thoughts:

There should probably be a global PS variable for setting the encoding to use
This global variable might be different across platforms

Joey Aiello (@joeyaiello)

Joey Aiello (@joeyaiello) do we need to fix this as a bug? A mixed-encoding file is just... wrong. It's bad input. I'm not really sure what Get-ChildItem should even do about it.

Joey Aiello (@joeyaiello)

CAT works so why would it be OK for Get-ChildItem to not work?
"If you want things to work avoid the PS tools?"

On Wed, Apr 13, 2016 at 11:06 AM, Andy Schwartzmeyer <
[email protected]> wrote:

Joey Aiello (@joeyaiello) https://github.com/joeyaiello do we need to fix this as a
bug? A mixed-encoding file is just... wrong. It's bad input. I'm not really
sure what Get-ChildItem should even do about it.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#707 (comment)

Linux uses UTF-8 without a BOM by default
.NET's Core implementation does not use a BOM
Open question: what's the default encoding on OS X?
PS does add a BOM to outputted files (we need to investigate exactly where)

Linux UX sync agrees that "doing the right thing" means making sure that PowerShell's default behavior works best with the tooling and ecosystem where it exists. Therefore, we shouldn't break the existing behavior on Windows, and we should change the behavior on Linux. To "do the right thing" on Linux, we have to make sure we don't ever add a BOM on Linux.

We have to test the non-BOM UTF-8 file generated on Linux can be read properly on Windows.

These changes would need to be made in the following cmdlets:

Out-File
Set-Content

We should also create an environment OR PS variable like $DefaultFileEncoding or $FileEncoding that changes the default behavior of the above cmdlets.

i validated that .NET does put a BOM in the file (in a similar way that we open files), and that our cmdlets do not do anything specific to add the BOM.

PS# $tfile = "$PWD\tfile.txt"
PS# $utf32enc = [text.encoding]::UTF32
PS# $fw = [io.filestream]::New($tfile,([io.filemode]::CreateNew))
PS# $sw = [io.streamwriter]::New($fw,$utf32enc)
PS# $sw.flush()
PS# $sw.dispose()
PS# $fw.dispose()
PS# format-hex $tfile

           Path: F:\e\rs1d\admin\monad\src\tfile.txt

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000000   FF FE 00 00 00 00 00 00 00 00 00 00 00 00 00 00  .þ..............

It is the case that if you change to utf8 encoding the appropriate BOM is written in the file,

James Truher (@JamesWTruher)

James Truher (@JamesWTruher) the results are different on Linux with .NET Core:

Out-File

PowerShell, on Linux, prepends a BOM when using UTF-8 encoding:

> "hello world" | Out-File -Encoding utf8 utf8
> file utf8
utf8: UTF-8 Unicode (with BOM) text

.NET Encoding

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;

namespace ConsoleApplication
{
    public class Program
    {
        public static void Main(string[] args)
        {
            String[] lines =
                {
                    $"BufferHeight: {Console.BufferHeight}",
                    $"BufferWidth: {Console.BufferWidth}",
                    $"WindowHeight: {Console.WindowHeight}",
                    $"WindowWidth: {Console.WindowWidth}",
                    $"LargestWindowHeight: {Console.LargestWindowHeight}",
                    $"LargestWindowWidth: {Console.LargestWindowWidth}",
                    $"IsErrorRedirected: {Console.IsErrorRedirected}",
                    $"IsOutputRedirected: {Console.IsOutputRedirected}",
                    $"IsInputRedirected: {Console.IsInputRedirected}",
                    ""
                };

            using (var stream = File.CreateText("Default.log"))
            {
                foreach (var line in lines) { stream.WriteLine(line); }
            }

            var encodings = new Dictionary<String, Encoding>()
                {
                    { "UTF8-Default.log", new UTF8Encoding() },
                    { "ASCII.log", new ASCIIEncoding() },
                    { "UTF8-ExplicitBOM.log", new UTF8Encoding(true) },
                    { "Unicode.log", new UnicodeEncoding() },
                };

            foreach (var encoding in encodings)
            {
                using (var file = new FileStream(encoding.Key, FileMode.Create))
                using (var stream = new StreamWriter(file, encoding.Value))
                {
                    foreach (var line in lines) { stream.WriteLine(line); }
                }
            }
        }
    }
}

$ dotnet run
$ for i in *.log; do file $i; done
ASCII.log: ASCII text
Default.log: ASCII text
Unicode.log: Little-endian UTF-16 Unicode text
UTF8-Default.log: ASCII text
UTF8-ExplicitBOM.log: UTF-8 Unicode (with BOM) text

Obviously that text was from other experiments 😄 If you include some non-ASCII text:

            String[] lines =
                {
                    "Normal ASCII text",
                    "你好"
                };

You get:

ASCII.log: ASCII text
Default.log: UTF-8 Unicode text
Unicode.log: Little-endian UTF-16 Unicode text
UTF8-Default.log: UTF-8 Unicode text
UTF8-ExplicitBOM.log: UTF-8 Unicode (with BOM) text

Which makes sense, as .NET is properly giving up on encoding non-ASCII characters for ASCII, but encodes as UTF-8 as it's supposed to:

$ cat ASCII.log
Normal ASCII text
??

$ cat UTF8-Default.log
Normal ASCII text
你好

.NET Core's UTF8Encoder is specifically set to not emit a BOM by default. (Link obtained from the API browser.)

I think we're just using https://github.com/dotnet/coreclr/blob/master/src/mscorlib/src/System/Text/Encoding.cs/#L1542 which passes true so the behavior should be the same on Linux.

OMG the fact that UTF8Encoding and Encoding.UTF8 have different explicit default behaviors is absurd.

Eric Erhardt (@eerhardt)

I just checked with Eric Erhardt (@eerhardt), and he confirmed this is the same behavior as .NET 4.6.1. The two different "default" UTF-8 constructors differ in BOM usage.

My humble opinion shares some other opinions above. Default for Out-File should be UTF8 without BOM in every system.

For 6.0.0 release we'll be doing a reduced scope version of this #4878

James Truher (@JamesWTruher)

James Truher (@JamesWTruher) could you clarify here what wasn't achieved by the implementation that we're shipping in 6.0?

Jeffrey Snover (@jpsnover)

The original issue from Jeffrey Snover (@jpsnover) no longer repros with RC2 with James Truher (@JamesWTruher)'s changes. I believe the change from Jim is that we default $OutputEncoding to utf-8 NoBOM instead of ASCII.

Navigation Menu

Repository navigation

Resolve UTF-8, UTF-16, ASCII inconsistencies #707

Out-File

.NET Encoding

33 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Resolve UTF-8, UTF-16, ASCII inconsistencies #707

Description

Activity

joeyaiello commented on Mar 24, 2016

andyleejordan commented on Apr 13, 2016

jpsnover commented on Apr 14, 2016

joeyaiello commented on May 16, 2016

JamesWTruher commented on May 16, 2016

andyleejordan commented on May 16, 2016

Out-File

.NET Encoding

andyleejordan commented on May 16, 2016

andyleejordan commented on May 16, 2016

lzybkr commented on May 16, 2016

andyleejordan commented on May 16, 2016

andyleejordan commented on May 16, 2016

33 remaining items

regs01 commented on May 16, 2017

SteveL-MSFT commented on Sep 30, 2017

joeyaiello commented on Dec 19, 2017

SteveL-MSFT commented on Jan 2, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions