Inconsistent handling of native command stderr

I discovered recently that the handling of stderr from native commands isn't being dealt with consistently between Windows PowerShell and Windows PowerShell ISE.

Steps to reproduce

$errorActionPreference="stop"; try { cmd /c nosuchexe } catch { 'dang!' }

Actual ISE behavior (expected)

dang!

Native commands writing to stderr while $errorActionPreference="stop" generate a catchable NativeCommandError which encapsulates the original error message in a ErrorRecord. It is easy to capture, analyze, and reword error messages in context.

Actual CLI behavior

'nosuchexe' is not recognized as an internal or external command,
operable program or batch file.

No NativeCommandError is generated, no matter how $errorActionPreference is set. The stderr from the native command is piped straight through to the default process stderr.

This behavior is a serious loss of functionality, leading some to believe that PowerShell can't distinguish between stderr and stdout. Lots of community confusion generated because of the discrepency. See https://stackoverflow.com/questions/44488202/powershell-streaming-output/44491420#44491420, among others.

Analysis

The difference isn't just the ISE, I think it is related to PowerShell remote runspaces! The full info on the caught error record (in the ISE) demonstrates the underlying exception is a RemoteException like this:

PS C:\Users\Burt_> $errorActionPreference="stop"; try { cmd /c nosuchexe } catch { Format-List * -force -InputObject $_ }



PSMessageDetails      : 
Exception             : System.Management.Automation.RemoteException: 'nosuchexe' is not recognized as an internal or external 
                        command,
TargetObject          : 'nosuchexe' is not recognized as an internal or external command,
CategoryInfo          : NotSpecified: ('nosuchexe' is ...ternal command,:String) [], RemoteException
FullyQualifiedErrorId : NativeCommandError
ErrorDetails          : 
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}

Environment data

Both powershell.exe and powershell_ise.exe report the same version information:

Name                           Value
----                           -----
PSVersion                      5.1.15063.296
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.15063.296
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Other community confusion probably due to this inconsistency:

I haven't evaluated each one, but I believe this has been a pain point for years.

I believe #3415 is a way to address part of your concern. This repo isn't for reporting Windiws PowerShell nor ISE issues, that would be Uservoice.

Steve Lee (@SteveL-MSFT)

Thanks Steve Lee (@SteveL-MSFT), here's my Uservoice link: https://windowsserver.uservoice.com/forums/301869-powershell/suggestions/19582069-inconsistant-handling-of-native-command-stderr. Unfortunately I don't think this will get much attention as many people have dismissed it as something PowerShell can't handle.

As to cross-platform PowerShell, in this case, I believe that the ISE provides the more desirable behavior, allowing easy access to trapping stderr. I believe such behavior is also available using PS remoting, but haven't tested it. Without this behavior (or something similar) it puts cross-platform PowerShell at a disadvantage from other environments which clearly separate stderr and stdout.

The parallel to set -eu is an interesting one. However the assertions you made in #3415 $errorActionPreference not working for native commands is NOT always true, nor documented as a feature. I suggest it's an (undesirable) quirk of the way local native commands are implemented.

The MSDN public API docs say:

Defines the Action Preference options. These options determine what will happen when a particular type of event occurs. For example, setting shell variable ErrorActionPreference to "Stop" will cause the command to stop when an otherwise non-terminating error occurs.

The difference between the ISE and the CLI should be fixed by 161940c - but that change would need to be back ported to Windows PowerShell.

With that change, I expect (but have not confirmed) that you'll get consistency, but not the behavior you desire.

I can say with certainty that tools do write to stderr under normal conditions, e.g., some tools write their logon/logo message to stderr to make it easy to redirect that message to /dev/null or nul, and that's why the default is to not report stderr output as an error in the cli.

Jason Shirk (@lzybkr)

Thanks, Jason Shirk (@lzybkr) . That PR seems to further codify the undesirable separation between native command and cmdLet by adding a Boolean flag to _WriteErrorSkipAllowCheck. I suspect you are right, and it's not the behavior I want.

Steve Lee (@SteveL-MSFT)

Steve Lee (@SteveL-MSFT): The ISE is incidental, the problem does affect PS Core:

When ~~remoting~~ a host other than the console host is involved (which includes the ISE and remoting contexts, even when invoked from the console), there is a fundamental difference in how external-utility calls are being handled; there is no obvious reason for this difference, and the inconsistency is problematic.
~~The remoting scenarios actually integrates much better with PowerShell's error handling when calling external utilities, whereas the current non-remoting behavior is unhelpful.~~

Undoubtedly, changing the behavior (at least by default) would be a high-profile breaking change.

While there is a documented difference in how remoting handles errors (from Invoke-Command's help),

Errors that result from the command that Invoke-Command runs are included in the command results. Errors that would be terminating errors in a local command are treated as non-terminating errors in a remote command.

, this difference does not explain the current behavior with external utilities.

Let me summarize Burt's findings for contrast:

(Non-remoting) console-hosts invocations:

& {
  $shell, $option = (('sh', '-c'), ('cmd', '/c'))[$env:OS -eq 'Windows_NT']; 
  $Error.Clear(); $ErrorActionPreference = 'Stop'
  try {  $null = & $shell $option nosuch } catch { 'dang!' }
  $?; $Error.Count
}

Output:

/bin/sh: nosuch: command not found   # (stderr printed straight to console in default color (Unix error message)
False   # no exception triggered, $? reflects nonzero exit code
0     # nothing was recorded in $Error

Stderr output is passed straight through to the console, not printed in red.
Since nothing is written to PowerShell's error stream - and therefore nothing getting captured in $Error - using $ErrorActionPreference = 'stop' with try / catch has no effect.
The only integration with PowerShell's error handling is that $? is set based on whether the exit code is zero ($True) or not ($False) (and the exit code is reflected in $LASTEXITCODE), but - unlike in the remoting scenarios - stderr output alone does not cause $? to be set to $False.

CAVEAT: The behavior changes fundamentally when stderr is redirected (becomes like the remoting-scenario behavior(!)): see #4002

In short:

Without using explicit redirection (2>&1 , 2>file), there is no way to detect the presence of or inspect stderr output.
Unlike PowerShell errors, which print in red, stderr output blends in with stdout output in the console (though it wouldn't be captured or redirected as part of the success stream, given that it prints straight to the console (in the absence of explicit redirection of the error stream)).

Non-console-host invocations, including in the ISE and in remoting contexts:

There, calls interact differently with PowerShell's error handling, with stderr output treated as if non-terminating errors had been reported; specifically:

# Run in *elevated* session, with *remoting set up*.
Invoke-Command -Computer . {
  $shell, $option = (('sh', '-c'), ('cmd', '/c'))[$env:OS -eq 'Windows_NT']; 
  $Error.Clear(); $ErrorActionPreference = 'Stop'
  try {  $null = & $shell $option nosuch } catch { 'dang!' }
  $?; $Error.Count
}

Output:

dang!  # An exception was triggered and caught - stderr output was considered a non-terminating error
True  # ($? is $True, because try/catch reset it)
1  # The stderr line *was* recorded in $Error

Every stderr output line is output as an error record and also recorded in the automatic $Error collection.
As in non-remoting consoles, automatic variable $? is set to $False for nonzero exit codes, but also when any stderr output was produced - independently of the exit code.
Combining $ErrorActionPreference = 'Stop' with a Try / Catch handler ONLY triggers the catch handler if there's at least 1 line of stderr output, and not with a nonzero exit code, even though the latter is the only true failure signal.

In short:

In ~~remoting scenarios~~ non-console-host invocations, stderr output behaves like non-terminating errors (as cmdlets would report).
However, there is no complementary mapping of nonzero exit codes to terminating errors, which means that $ErrorActionPreference = 'Stop' with a Try / Catch doesn't catch a quietly failing utility (one that signals failure by its exit code only).
- Explicitly testing $LASTEXITCODE (or $?, if no try / catch was involved) is still needed.
- A possible solution is to make the behavior similar to that of .NET method calls (whose exceptions are treated as non-terminating errors without try / catch, but are caught if enclosed in try / catch - although that could be considered a historical wart itself); translated to this scenario:
  - ~~As before: outside of try / catch, merely reflect a nonzero exit code in $LASTEXITCODE, without further action.~~
  - ~~New: inside of try / catch, throw an exception if the exit code is nonzero.~~

I found 5 random complaints about the ISE or remoting reporting errors with native executables where there was no error:

https://stackoverflow.com/questions/31449220/powershell-ise-wrongly-interprets-openssl-exe-normal-output-as-error
https://stackoverflow.com/questions/10856609/why-is-powershell-ise-showing-errors-that-powershell-console-does-not-show
https://stackoverflow.com/questions/12537254/tell-powershell-ise-to-not-send-stderr-to-write-error
https://stackoverflow.com/questions/2095088/error-when-calling-3rd-party-executable-from-powershell-when-using-an-ide
https://stackoverflow.com/questions/18380227/psexec-throws-error-messages-but-works-without-any-problems

This is why powershell.exe behaves as it does - it is common for tools to use stderr and not report errors, and automation is broken if we assume otherwise. If you follow the links, you'll see those tools are common as well - git, hg, openssl, and 2 SysInternals utilities: psexec, and du.

Jason Shirk (@lzybkr)

Jason Shirk (@lzybkr) I agree with you that we can't rely on stderr to be an Error, but it seems we have difference in behavior between local and remote PowerShell and that we should fix

cc @PowerShell/area-remoting

This is not remoting specifc, nor ISE specific. There was a bug in the native command processor, which was worked around in the console host, but not in the "default host" which is used by remoting, nor was it worked around in the ISE host. I referenced the real fix above - 161940c

You can see the bug in an ordinary runspace:

$ps = [powershell]::Create().AddScript('cmd /c nosuch')
$ps.Invoke()
$ps.Streams.Error

In Windows PowerShell, you'll get:

cmd : 'nosuch' is not recognized as an internal or external command,
At line:1 char:1
+ cmd /c nosuch
+ ~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: ('nosuch' is not...ternal command,:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

operable program or batch file.

In PowerShell Core, you'll get:

'nosuch' is not recognized as an internal or external command,
operable program or batch file.

I've found an apparently undocumented workaround that captures stderr into $Error independent of host. Its actually a bit surprising, but might serve as a inspiration for developing a supported solution for native commands that fits better with PowerShell's error handling design, but doesn't present the breaking change.

If you add 2>$null to a native command, it captures stderr into $Error! I haven't dug into how this works, but it seems consistent across hosts. Further detail here.

Now I don't particularly like the syntax, but stumbled on it when I tried 2> which generated Missing file specification after redirection operator. I had hoped that something like 2>$a might act like -ErrorVariable a, but that's wishful thinking. Resorted to $null as a placeholder to get around the syntactic requirement for redirection, and got behavior (including trapping if $errorActionPreference='stop') that's pretty much what I had hoped for.

My poking around was inspired by the 2>&1 syntax helps out for cases where a command uses stderr for non-error output, and looking for a complementary notation. To extend this concept further and support full -ErrorVariable a semantics, it might be possible to support a redirection syntax like 2>&a (getting rid of the dollar sign is a good thing.) Since redirection is implemented by the shell, and not passed to the child's argv, it's transparent to the native utility.

If a further similar redirection-like syntax were able to add -ErrorAction semantics, it would put native command execution on a similar footing to cmdlet execution. I was thinking perhaps something like 2>+Stop to give just one native command terminating stderr semantics.

Also, DefaultHost is used in other contexts, e.g. Start-Job. Will the 161940c fix be a breaking change for them as well?

Jason Shirk (@lzybkr)

Jason Shirk (@lzybkr): Thanks for finding the real cause - I've cleaned up my previous post accordingly.

it is common for tools to use stderr and not report errors, and automation is broken if we assume otherwise

You're right: external utilities have no choice other than to use stderr for anything that's not data, so stderr is frequently used even for non-error messages, and the only true indicator of an external utility's success is its exit code.
My initial thought was that it would be harmless to treat stderr lines as non-terminating PS errors, as they'd just collect in $Error without stopping the script (and how they should be formatted for display is a secondary issue).

But I can definitely see how you do NOT want your script to abort with $ErrorActionPreference = 'Stop' in effect just because an external utility happens to write to stderr, and how $? should be set solely based on the exit code.

In an ironic twist, that is precisely what happens when you explicitly want to suppress stderr output currently (or redirect it to a file) - see #4002:

 & { $ErrorActionPreference = 'stop'; cmd /c nosuch 2>$null; $? } # ABORTS

As Burt Harris (@BurtHarris) has discovered, 2>$null doesn't actually just mean "discard that stream's output" (and 2>file doesn't just mean "save that stream's output to a file"), but it currently also means "quietly record the output lines as error records in $Error", which is:

(a) completely unexpected
(b) badly interacts with $ErrorActionPreference = 'stop', as demonstrated (writing to $Error apparently is tantamount to a non-terminating error)

However, as Burt Harris (@BurtHarris) mentions, there is value in making stderr output available to subsequent commands, at least on demand, but 2>$null is clearly not the way to do it.

For instance, being able to inspect stderr input is important for ill-behaved external utilities that, instead of signaling success via their exit code, write to stderr only.

So, as Burt Harris (@BurtHarris) suggests, having something analogous to -ErrorVariable would be helpful.
While surfacing that via > is a natural candidate, it makes it difficult to separate the immediate output behavior from the record-in-a-variable behavior, unless some presumably awkward compound syntax is devised.

An -ErrorAction analog might also be nice - which raises similar syntax questions - but, based on the discussion above, its semantics should be tied to the exit code, not to stderr output.

Joey Aiello (@joeyaiello)

Reviewing this issue for the first time in over a year. Thanks for the continued attention to it, please forgive me if I'm not fully up-to-date.

Joey Aiello (@joeyaiello) said

I still believe strongly that stderr is not a reasonable indicator that a real error has taken place.

I would agree if we changed the word "reasonable" to "sufficient". There are too many different practices in command-line native commands for it to be considered "sufficient". The tricky part is that the meaning of stderr output is both context and content dependent. But a script knows (or defines) the context, and what content might indicate a real error or meaningful information stream event.

Has any thought been put into using a preference variable to let a script customize the treatment of stderr handling? Using $ErrorActionPreference is problematic, but how about something like $StderrActionPreference?

I'm not sure if a simple true/false or ignore/continue/error/abort is flexible enough, but perhaps some sort of regex matching approach, similar to how IDEs are adapted to different compile tools (like a VSCode task problemMatcher.) [This line-of-thought is perhaps informed by thought years ago about a composable text-to-object mapping facility for PS, but at the time I was thinking bout stdout text.]

Burt Harris (@BurtHarris)

Burt Harris (@BurtHarris) there is an existing RFC that the @PowerShell/powershell-committee supports for handling exit codes. Stderr isn't reliable to detect errors as native tools use it as for anything that isn't intended to be stdout. Exit codes have their own problems where non-zero doesn't indicate an error, but most of the time, a non-zero exit code is an error case.

Changing stderr to be information stream is a breaking change in that redirection of stream 2 would not work as expected for native commands. Unfortunately, I think we are stuck with this design.

However, I agree with Michael Klement (@mklement0)'s proposal that we special case NativeCommandError error records from triggering $ErrorActionPreference as a way to resolve this specific issue.

@PowerShell/powershell-committee discussed this and agrees for 7.1 we should have NativeCommandError error records not trigger $ErrorActionPreference although a breaking change, but is likely bucket 3

Glad to see that this is getting tackled:

I see that it is currently implemented as an experimental feature, PSNotApplyErrorActionToStderr
Unfortunately, it doesn't address the $? issue (which I snuck in later above, based on Redirection of stderr output from external programs sets $? invariably to $false, no longer reflecting whether $LASTEXITCODE equals 0 #10512 - I should have made it explicit)

That is, even with the experimental feature enabled the following doesn't work as intended:

# On Unix
PS> sh -c 'ls nosuch; :' 2>$null; $?
False # !! Should be $true

Since the exit code of the sh command is 0, $? should be $true - irrespective of the presence of stderr output, whether redirected or not.

This matters primarily with respect to && and ||:

# On Unix
PS> sh -c 'ls nosuch; :' 2>$null || 'why did I get here?'
why did I get here?

Just so we don't lose track of this: see #13393

Steve Lee (@SteveL-MSFT)

I was REALLY happy to see this was addressed. However, although the problem seemed to be fixed in 7.1.0-preview.7, it has regressed in 7.1.0-rc.1. :'( Was this intentional? I didn't see anything in release notes about it; did I miss it? I hope that it can be re-fixed... Steve Lee (@SteveL-MSFT) Michael Klement (@mklement0)

(False alarm: it's still an experimental feature ('PSNotApplyErrorActionToStderr'), and I didn't realize it was on by default in prev7, and back to off in rc1.)

Release 2.1.0: fix PowerShell 5.1 stderr bug, remove Outlook, HUD upd…

Navigation Menu

Repository navigation

Inconsistent handling of native command stderr #3996

Steps to reproduce

Actual ISE behavior (expected)

Actual CLI behavior

Analysis

Environment data

43 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Inconsistent handling of native command stderr #3996

Description

Steps to reproduce

Actual ISE behavior (expected)

Actual CLI behavior

Analysis

Environment data

Activity

BurtHarris commented on Jun 13, 2017

SteveL-MSFT commented on Jun 13, 2017

BurtHarris commented on Jun 13, 2017

BurtHarris commented on Jun 13, 2017

lzybkr commented on Jun 13, 2017

BurtHarris commented on Jun 13, 2017

mklement0 commented on Jun 13, 2017

lzybkr commented on Jun 13, 2017

SteveL-MSFT commented on Jun 13, 2017

lzybkr commented on Jun 13, 2017

BurtHarris commented on Jun 14, 2017

BurtHarris commented on Jun 14, 2017

mklement0 commented on Jun 14, 2017

43 remaining items

BurtHarris commented on Jul 31, 2020

SteveL-MSFT commented on Jul 31, 2020

SteveL-MSFT commented on Aug 5, 2020

mklement0 commented on Aug 7, 2020

mklement0 commented on Aug 10, 2020

jazzdelightsme commented on Oct 20, 2020

jazzdelightsme commented on Oct 21, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions