Don’t parse the pipeline as text when it is directed from an EXE to another EXE or file. Keep the bytes as-is.

Currently PowerShell parses STDOUT as string when piping from an EXE, while in some cases it should be preserved as a byte stream, like this scenario:

curl.exe http://whatever/a.png > a.png

or

node a.js | gzip -c > out.gz

Affected patterns include: native | native, native > file and (maybe) cat file | native.

sergei (@vors)

sergei (@vors) Jason Shirk (@lzybkr)
The current NativeCommandProcessor breaks:

LF line endings.
Non-ASCII text within UTF-8 without BOM header.
Binary file redirects (like curl.exe’s output).
> layouts text into 80 columns by default.

Maybe add a cmdlet/operator to call native command and get its raw output (as a byte array / stream?), something like this:

# Consider ^& operator is an alias for Get-CommandRawOutputStream; this is just an example syntax
$output = ^& curl.exe http://whatever/a.png # $output now is a byte array or stream
$output > C:\Temp\file.png # file.png now is a valid image file

# This should be valid, too:
^& curl.exe http://whatever/a.png > C:\Temp\file.png

This opens an opportunity for some additional usage patterns (you can put this raw content into variables, and pipe raw content from native commands to managed cmdlets).

But maybe we could add a special kind of redirection operator (like 2>&1, 3>&1, *>&1 we already have), something like this (where %>&1 is a new redirection operator that redirects command "raw output" without processing it as a string):

$output = curl.exe http://whatever/a.png %>&1
$output > C:\Temp\file.png

# Or even this:
curl.exe http://whatever/a.png %> C:\Temp\file.png # which is just awesome

Overall: I don't think that this kind of redirection should be tied to only native commands or some limited list of usage patterns (e.g. native | native).

Friedrich von Never (@ForNeVeR)

Friedrich von Never (@ForNeVeR) My proposal is that:

For native | native, keep the bytes as-is. This is already purposed by sergei (@vors).
For ps | native, add a set of cmdlets which encodes PS objects into bytes, perhaps ps | encode-text utf-8 | native.
For native | ps, we can use the type system to identify whether a cmdlet accepts “raw input”. For cmdlets like out-file or maybe decode-text, it will keep the bytes from native, and other cmdlets will use the parsed string as its input.

Belleve (@be5invis)

Belleve (@be5invis) okay, it seems like this proposal also supports all the relevant use cases I can imagine.

Shouldn't this open up an RFC since this is a breaking change (changes the observed behaviour)?

A workaround for this is to provide a cmdlet that stores the content in a temporary file. A working example is Use-RawPipeline in PowerShell Gallery. The current implementation is to store the file, but it could also be streamlined so that the file doesn't have to be stored.

sergei (@vors)

See also #559, where this appears to be actively discussed and worked on by sergei (@vors) on the PowerShell team.

wooooooooooo this works: zstd --decompress Digest-BLAKE-0.05.tar.zst --stdout | gzip -9 | gzip -dc | tar -axvf -

.\\Digest-BLAKE-0.05/
.\\Digest-BLAKE-0.05/BLAKE.xs
.\\Digest-BLAKE-0.05/Changes
.\\Digest-BLAKE-0.05/ex/
.\\Digest-BLAKE-0.05/ex/benchmark.pl
.\\Digest-BLAKE-0.05/lib/
...

For me though, with this feature enabled, you can't actually redirect stdout to a file, it always seems to go to the console with native commands followup bug filed though #19836

I know this thread is rather old, but I have some issues with the current native to native implementation.

For some reason, it is very slow compared to the command processor (cmd.exe) pipeline. No matter what I try, this reaches 45 MB/s for me in the peak when reading from one drive and writing the exact same data to another drive (it doesnt only affect that, reading and writing dummy data was just to make sure the CPU is not the bottleneck). The command processor does it at full capacity of my drives, it gets the data through at 180 MB/s (the max my HDDs support)
Executables to receive via stdin cant be expressed with a full path or am I missing something? At the beginning of the pipeline, the executable may be written with the full path without any issues. But the second executable (or later ones) can only be called if the exe file is either in the current working directory or in the path variable. I have not found a way to chain multiple executables together, all with their full path to the exe file.

Are there known issues with the speed? How can I chain multiple native commands, each with its full path?

I don’t know about the speed, but I can call full paths just fine, so you may need to describe your issue in more detail:

> echo.exe yo | D:\msys64\usr\bin\cat.exe
yo
> echo.exe yo | & D:\msys64\usr\bin\cat.exe
yo

I don’t know about the speed, but I can call full paths just fine, so you may need to describe your issue in more detail:
> echo.exe yo | D:\msys64\usr\bin\cat.exe
yo
> echo.exe yo | & D:\msys64\usr\bin\cat.exe
yo

You're right, it works. Sigh, I had the parameters in quotation marks.

Regarding speed, I did the following

fsutil.exe file createnew test_10gb 10737418240 //Create a test 10 GB file
C:\...\Cygwin64\bin\cat.exe .\test_10gb | & "C:\...\Cygwin64\bin\split.exe" -b500MB - F:\Users\...\tmp\split_test.

F is a HDD drive here. The exact same command (except for the ampersand) maxes out my HDDs when I run it in cmd.exe.

What version of powershell? What version of windows? I would do something more straightforward like:
Here is older powershell without proper pipelining

dd if=dotnet-sdk-8.0.412-win-x64.exe bs=1MB | dd of=NUL bs=1MB
224+1 records in
224+1 records out
224233512 bytes (224 MB, 214 MiB) copied, 16.5476 s, 13.6 MB/s

with pipeline correct:

dd if=dotnet-sdk-8.0.412-win-x64.exe bs=1MB | dd of=NUL bs=1MB
224+1 records in
224+1 records out
224233512 bytes (224 MB, 214 MiB) copied, 0.352218 s, 637 MB/s

PS >dd if=dotnet-sdk-8.0.414-win-x64.exe bs=1MB | dd of=NUL bs=1MB
220+1 records in
220+1 records out
220218536 bytes (220 MB, 210 MiB) copied, 2.9428 s, 74.8 MB/s
0+38695 records in
0+38695 records out
220218536 bytes (220 MB, 210 MiB) copied, 2.94187 s, 74.9 MB/s
PS >

in

PS >$PSVersionTable | fl

Name  : PSVersion
Value : 7.5.3

Name  : PSEdition
Value : Core

Name  : GitCommitId
Value : 7.5.3

Name  : OS
Value : Microsoft Windows 10.0.26100

Name  : Platform
Value : Win32NT

Name  : PSCompatibleVersions
Value : {1.0, 2.0, 3.0, 4.0…}

Name  : PSRemotingProtocolVersion
Value : 2.3

Name  : SerializationVersion
Value : 1.1.0.1

Name  : WSManStackVersion
Value : 3.0

More specifically, I'm on 26100.6584.
In cmd.exe, I get

dd if=dotnet-sdk-8.0.414-win-x64.exe bs=1MB | dd of=NUL bs=1MB
220+1 records in
220+1 records out
220218536 bytes (220 MB, 210 MiB) copied, 1.4477 s, 152 MB/s
0+24811 records in
0+24811 records out
220218536 bytes (220 MB, 210 MiB) copied, 1.48674 s, 148 MB/s

At those speeds I don't think your issue is related to this, looks like you are likely maxing your drive read speed there.

Florian (@dt-flo)

you are likely maxing your drive read speed there.

They’re getting a 2× speed in cmd.exe on that same drive.

Florian (@dt-flo) Do make sure it’s not just due to disk caching, though. (That is, if you run the command first in PS and then in cmd.exe, some of the data may still be cached in memory and not actually be read from disk during the second test at all.)

I tried running it in PWSH again today:

dd if=dotnet-sdk-9.0.305-win-x64.exe bs=1MB | dd of=dotnet_out bs=1MB
226+1 records in
226+1 records out
226741024 bytes (227 MB, 216 MiB) copied, 2.60769 s, 87.0 MB/s
0+51725 records in
0+51725 records out
226741024 bytes (227 MB, 216 MiB) copied, 2.65083 s, 85.5 MB/s

Switched to cmd.exe

dd if=dotnet-sdk-9.0.305-win-x64.exe bs=1MB | dd of=dotnet_out bs=1MB
226+1 records in
226+1 records out
226741024 bytes (227 MB, 216 MiB) copied, 1.14248 s, 198 MB/s
0+27754 records in
0+27754 records out
226741024 bytes (227 MB, 216 MiB) copied, 1.16856 s, 194 MB/s

Switched back to PWSH and tried twice:

dd if=dotnet-sdk-9.0.305-win-x64.exe bs=1MB | dd of=dotnet_out bs=1MB
226+1 records in
226+1 records out
226741024 bytes (227 MB, 216 MiB) copied, 2.3447 s, 96.7 MB/s
0+52359 records in
0+52359 records out
226741024 bytes (227 MB, 216 MiB) copied, 2.37055 s, 95.6 MB/s

dd if=dotnet-sdk-9.0.305-win-x64.exe bs=1MB | dd of=dotnet_out bs=1MB
226+1 records in
226+1 records out
226741024 bytes (227 MB, 216 MiB) copied, 2.36039 s, 96.1 MB/s
0+52114 records in
0+52114 records out
226741024 bytes (227 MB, 216 MiB) copied, 2.387 s, 95.0 MB/s

That was in Windows with Cygwin. I thought in order to exclude a possible influence from my AV Bitdefender (even if it's "turned off"), I tried it in Linux on the same drive.

The result is as follows:
Bash:

flo@flo-VMware7-1:/tmp$ dd if=dotnet-sdk-9.0.305-win-x64.exe bs=1MB | dd of=dotnetout bs=1MB
226+1 records in
226+1 records out
226741024 bytes (227 MB, 216 MiB) copied, 0.988518 s, 229 MB/s
0+3588 records in
0+3588 records out
226741024 bytes (227 MB, 216 MiB) copied, 0.98907 s, 229 MB/s

PowerShell 7:

flo@flo-VMware7-1:/tmp$ pwsh
PowerShell 7.5.3
PS /tmp> dd if=dotnet-sdk-9.0.305-win-x64.exe bs=1MB | dd of=dotnetout bs=1MB
226+1 records in
226+1 records out
226741024 bytes (227 MB, 216 MiB) copied, 1.68791 s, 134 MB/s
0+106333 records in
0+106333 records out
226741024 bytes (227 MB, 216 MiB) copied, 1.93509 s, 117 MB/s
PS /tmp>

Even if it was in disk cache, PowerShell should be as quick as bash this time because I ran it in PWSH after I ran it in Bash, right?

Edit: I should have said that I tried it on an SSD today, that's why it's faster than yesterday.

I have made another test on another device (same Windows build as the first test machine). This has a built-in NVMe SSD. I have installed a fresh debian 12 with PowerShell 7.5.3 in Hyper-V. The results were the following:

I have now tested this on a third device (this device runs Ubuntu 24 LTS as main operating system, so no virtualization) and again, PowerShell is significantly slower than Bash.
Can anyone confirm that PowerShell 7.5 is way slower compared to any other shell (e.g. Bash on Linux, cmd.exe on Windows) on their device(s) as well?

Should I open a new issue?

Because this actually prevents me from making PowerShell my only (used) shell on Linux systems and using PowerShell to chain multiple executable files.

Should I open a new issue?

You should definitely open a new issue because it is unrelated to this closed one.

Steve (@hez2010)

Re Steve (@hez2010)

Any updates? Seems that without this we cannot use gzip to compress data stream:
cat -AsByteStream a.js | gzip > a.gz
The data stream will be corrupted by the >.

This command works on every shell except PowerShell.

To make it work generally would bring insufferable performance loss (sometimes even non-halting). In addition, using Get-Content to write bytes into the pipeline is very expensive. It's like Java's ArrayList<Byte>.

The natural (wrong) consideration for cmdlet | native is as follows:

if all objects written by cmdlet to the output stream are byte objects, then pipe them unchanged to native;
otherwise, pipe all objects from cmdlet to out-string, then convert to bytes using some encoding, then pipe the converted bytes to native.

This, as a starter, would require cmdlet to run to completion before data can be piped to native, which violates the intention of pipes. Also, consider cmdlet and native communicating via a file (that is initially empty):

cmdlet writes 1 byte of 0x41 (ASCII A) to output stream, then read the file to see if it begins with byte 0x01. If so, it writes string "Hello". Otherwise, it writes another byte 0x41.
native reads 1 byte from stdin. If the byte is 0x41, then it writes 0x00 to the file. If the byte is 0x36 (ASCII 6), it writes 0x01 to the file.

The execution would never be correct, and a usual implementation does not halt. Whether or not binary pipe is used, must be determined before running the command.

Currently the operator < is reserved for future use. When it's implemented, it should provide the functionality of redirecting a single file as stdin. To obtain concatenated files, I think there needs to be some overhaul on pipe handling design.

To make it work generally would bring insufferable performance loss (sometimes even non-halting). In addition, using Get-Content to write bytes into the pipeline is very expensive. It's like Java's ArrayList<Byte>.

It does work now, piping byte or byte[] to a native command will send the bytes as is. This was more about enabling scenarios than performance, if perf is the goal then using .NET APIs instead would be recommended.

This, as a starter, would require cmdlet to run to completion before data can be piped to native, which violates the intention of pipes. Also, consider cmdlet and native communicating via a file (that is initially empty):

Currently you could pipe each byte one at a time if you wanted, but that would likely end up as worse performance. Note that piping to a native command is already not halting due to implementation detail.

Gotcha and I tried the current behavior and it's a gotcha for me (those are two different kinds of gotcha's lol). The previous example doesn't stand because it assumes the piping mechanism is either all stringification or all bytes (not the case) --- I assumed piping of byte array only applied if the upstream expression has a special form!

The actual behavior, which I think should be documented somewhere:

If the upstream is non-native and the downstream is native, then each object o in the upstream output is looked at individually, if !ReferenceEquals(o, null) && o.GetType() == typeof(byte) (i.e., it must be of exactly type [byte]) then it's written to the downstream as the byte, otherwise it goes through the stringification procedure.

Some examples that might be helpful in documentation:

# Previous meaning: stringify the bytes and send to native
# New      meaning: pipe a.txt into native
get-content -asbytestream a.txt | native

# To reinstate previous meaning, use
get-content -asbytestream a.txt -readcount 0 | native
# or
get-content -asbytestream a.txt | foreach { [int]$_ } | native

Navigation Menu

Repository navigation

Don’t parse the pipeline as text when it is directed from an EXE to another EXE or file. Keep the bytes as-is. #1908

99 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Don’t parse the pipeline as text when it is directed from an EXE to another EXE or file. Keep the bytes as-is. #1908

Description

Activity

be5invis commented on Aug 18, 2016

ForNeVeR commented on Aug 25, 2016

ForNeVeR commented on Aug 25, 2016

be5invis commented on Aug 25, 2016

ForNeVeR commented on Aug 26, 2016

GeeLaw commented on Sep 2, 2016

jhclark commented on Sep 6, 2016

99 remaining items

mitchcapper commented on Jun 22, 2023

dt-flo commented on Sep 17, 2025

astiob commented on Sep 17, 2025

dt-flo commented on Sep 17, 2025

mitchcapper commented on Sep 17, 2025

dt-flo commented on Sep 17, 2025

mitchcapper commented on Sep 17, 2025

astiob commented on Sep 17, 2025

dt-flo commented on Sep 18, 2025

dt-flo commented on Sep 18, 2025

dt-flo commented on Sep 26, 2025

hez2010 commented on Sep 26, 2025

GeeLaw commented on Dec 9, 2025

SeeminglyScience commented on Dec 9, 2025

GeeLaw commented on Dec 9, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions