Suggestion: Add redirection syntax that allows in-memory collection of stderr lines output by external utilities, akin to the -ErrorVariable common parameter

^{This was first proposed as part of a lengthy exchange in #3996, but I thought it deserved its own post.}

tl;dr

In order to collected stderr output (separately) from an external-utility call in memory, I suggest supporting the following new redirection syntax that allows capturing stderr lines in a variable, akin to the common -ErrorVariable parameter for cmdlets (the sample cmd command is designed to produce both stdout and stderr output):

# Wishful thinking: Collect stderr lines in given variable $stderrOutput via 2>&
$stdoutOutput = cmd /c ver '&' dir \nosuch 2>&stderrOutput

Instead of what you must currently do:

$stderrFile = New-TemporaryFile
$stdoutOutput = cmd /c ver '&' dir \nosuch 2>$stderrFile
$stderrOutput = Get-Content $stdErrFile
Remove-Item $stderrFile

The current behavior is to pass stderr output generated by external utility calls (e.g., to git) through to the console - stderr output is not recorded in $Error, which makes sense, given that many utilities write much more than just error information to stderr, and that the presence of stderr input doesn't imply actual errors.

Sometimes it is necessary to inspect stderr output, however, which is currently not easily accomplished:

You can redirect stderr output by redirecting PowerShell's error stream: 2>..., but you're faced with 2 options, neither of which is convenient:

Either: redirect stderr output to a file with 2>filename
Or: redirect stderr output into the success stream with 2>&1 to produce a combined stream from which you can later filter out the stderr lines by type ([System.Management.Automation.ErrorRecord]).

Therefore, I propose a new redirection syntax that allows capturing stderr lines in a variable, akin to the common -ErrorVariable parameter for cmdlets, so that:

foo.exe 2>&errs would be the external-utility-call equivalent of a (fictitious) Invoke-Foo -ErrorVariable errs 2>$null call, with errors getting collected in variable $errs in both cases.
Additionally, to allow stderr output to also be passed through while being collected:
foo.exe 2>&|errs could be the equivalent of Invoke-Foo -ErrorVariable errs

The suggestion is to use >&varName syntax, because & is already established as having into-a-different-target(-stream) semantics, yet currently only a digit is supported after the & (the index of a different PS stream).
(There is still ambiguity, given that $1 is a valid variable name, but I think that wouldn't be a real-world concern.)

Also, given that -ErrorVariable supports a + prefix to the variable name to indicate that the existing variable value should be appended to, this new syntax should support that too.

Environment data

PowerShell Core v6.0.0-beta.4

Rather than this approach which relies on syntactic element, it was possible to create redirection as follows:

PS> write-error bad 2>variable:var1

and

$a = 1..5
6..9 | write-error 2>>variable:a

while not as terse as a sigil, it required very little change. It also works with native executables ala:

PS> $result = /bin/ls -ld . sdfljksdj 2>variable:whoops
PS> $result
drwxr-xr-x  93 james  staff  2976 Jun 12 12:05 .
PS> $whoops
ls: sdfljksdj: No such file or directory

a further by-product is that now all streams may be redirected to a variable

$a = . { 
>> write-error bad
>> write-warning warning
>> write-verbose -verbose 'and here'
>> write-debug -debug 'yep, debug too'
>> write-information 'and info'
>> } 2>variable:e 3>variable:w 4>variable:v 5>variable:d 6>variable:i
PS> $e
Write-Error: bad
PS> $w
WARNING: warning
PS> $v
VERBOSE: and here
PS> $d
DEBUG: yep, debug too
PS> $i
and info

and append works too

PS> 1..3 | write-error 2>variable:e         
PS> /bin/ls -ld / lsjdflkjsdlj 2>>variable:e
drwxr-xr-x  23 root  admin  736 Jun  8 14:35 /
PS> $e
Write-Error: 1
Write-Error: 2
Write-Error: 3
ls: lsjdflkjsdlj: No such file or directory

I think that using variable:<name> is ok - I think it's somewhat natural for the PowerShell environment.
It required a relatively simple change to the engine and a small extension to set-variable. I can submit a PR if you think this is reasonable.

Just one question... is this generalizable to all PSProviders/PSDrives that support the Content API now?

Because IMO... it should be. 🙂

I think this has some potential to be very useful especially on Linux. There’s been a few times where I’ve needed to capture standard output, didn’t care to log it, But needed to perform certain actions based on what type of output it was.

James Truher (@JamesWTruher)

Thanks, James Truher (@JamesWTruher) - I like the idea, and personally I would definitely welcome a PR.

Follow-up questions:

To flesh out Rain Sallow (/u/ta11ow) (@vexx32) 's concern: Since this would amount to effectively supporting namespace variable notation as the redirection target, we should probably support all drives:
- Among the standard drives, aside from variable:foo, only env:Foo makes sense (writing to an environment variable), but conceivably there could be other providers whose drives could be meaningfully targeted.
The original proposal included an additional feature variation; do we need it, and, if so, what should the syntax be?
- Optionally also passing a stream through while capturing it (this would be like an in-command Tee-Object for any stream targeted).

I would like it better as a cmdlet we can pipe to before we make it an operator.

Dave Wyatt (@dlwyatt)

I would happily vote yes to making each and every one of the numbered streams generically redirectable to any content provider PSDrive path. That makes sense, it's always frustrating when something looks like it builds on the content providers and doesn't.

I think a version of Tee-Object that could capture streams beyond should be technically impossible, because only output goes to cmdlets in a pipeline, and it would be super confusing if there was some magic thing that looked like a cmdlet but could somehow intercept other output before it gets to the host.

As a use case / example: we love Dave Wyatt (@dlwyatt)'s PowerShellLogging module, but what it does is a dirty hack that shims the host. Unless you're going to make interfaces public so that sort of host interception logging would be possible in every host without using private methods, I don't need a cmdlet version of redirection, and I don't want weird new syntax for capturing without redirecting (that is, passthrough capturing)

P.S. Alias: is a content drive and works just as well as ENV: for holding strings... 🤯

I really like the ability to redirect to a psdrive, please make it work on any content providers drive, not just variable. I guess variable and env are the ones that makes sense currently, but just as jaykul says, having it look like it is using a content provider when it is not will just create a special case that will be hard to explain.

I love this. I re-implemented Start-Process to support this behavior so I could redirect stderr/stdout to a variable and return a System.Diagnostics.Process object with two additional properties that contain any output. Having this feature would simplify a lot of code for parsing command output. It's only a matter of time before my hack runs into a scenario it can't handle.

Joel Bennett (@Jaykul)

Joel Bennett (@Jaykul)

Agreed re Tee-Object cmdlet.

Unless you're going to make interfaces public so that sort of host interception logging

I don't think we should think of this in terms of hosts - the fact that external-program output is sent directly to the host by default should be considered an implementation detail. Instead, we want to be able to target streams:

The > operator already allows you to selectively redirect stderr output (which would by default go to the host, and with the proposal at hand we would now also support variables as targets (potentially among others).

The suggestion to enable optional pass-through functionality stems from a gap in functionality with respect to external programs and their stderr output, specifically.

For PowerShell commands you can use the -*Variable common parameters (e.g., -ErrorVariable), optionally via an aux. Invoke-Command call, to capture stream output with pass-through logic; that is, capturing stream output this way passes through by default, and you need to explicitly silence the target stream to suppress that.

This ability is currently lacking for external-program calls - you only have the choice between passing through to the host only, or redirecting and thereby invariably silencing the stream.

It is conceivable that you want to both see stderr output - and doing so in context, interleaved with stdout output is especially important when calling external programs - and capture it.

You can actually somewhat emulate this with Invoke-Command at the moment though it's both highly obscure and has major side effects, notably polluting the success output stream and changing the output sequencing:

# $e captures stderr output, while still passing it through - albeit invariably
# via the *success* output stream, and out of sequence
Invoke-Command { ls / nosuch 2>&1 } -ErrorVariable e

That said, if -ErrorVariable were able to capture stderr output as-is, then we arguably do not need a new syntax (I cannot speak to the technical feasibility):

# WISHFUL THINKING:
# Pass stderr output through, in output order, *and* capture it in variable $e
# (Currently, this records nothing in $e.)
Invoke-Command { ls / nosuch } -ErrorVariable e

P.S. Alias: is a content drive and works just as well as ENV: for holding strings..

Indeed; to spell it out: the proposal is to allow targeting drives of all those PS providers that implement the IContentCmdletProvider interface, which technically means the following standard providers:

PS> (Get-PSProvider | Where-Object { 
      $_.ImplementingType.GetInterfaces() -contains [System.Management.Automation.Provider.IContentCmdletProvider] 
    }).Name

Alias
Environment
FileSystem
Function
Variable

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

James Truher (@JamesWTruher)

A new redirection syntax was also proposed for #5184 what ever happened to this idea James Truher (@JamesWTruher)?

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.

Navigation Menu

Repository navigation

Suggestion: Add redirection syntax that allows in-memory collection of stderr lines output by external utilities, akin to the -ErrorVariable common parameter #4332

Environment data

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Suggestion: Add redirection syntax that allows in-memory collection of stderr lines output by external utilities, akin to the -ErrorVariable common parameter #4332

Description

Environment data

Activity

JamesWTruher commented on Jun 12, 2020

vexx32 commented on Jun 12, 2020

billkindle commented on Jun 12, 2020

mklement0 commented on Jun 12, 2020

KevinMarquette commented on Jun 12, 2020

Jaykul commented on Jun 12, 2020

SimonWahlin commented on Jun 13, 2020

qaswani commented on Jun 13, 2020

mklement0 commented on Jun 14, 2020

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

Jaykul commented on Nov 17, 2023

microsoft-github-policy-service commented on Nov 24, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions