Implement accommodations for Windows CLIs, notably batch files and misexec-style programs as part of experimental feature PSNativeCommandArgumentPassing

Travis Plunk (@TravisEz13)

Summary of the new feature/enhancement

A Guide to this Proposal:

The rest of this initial post details the motivation and the proposed implementation.
Easy-to-grasp examples of the proposed accommodations are in this comment.
Complementary examples of what won't work unless we implement these accommodations are in another comment.

As requested by Travis Plunk (@TravisEz13) in #14692 (comment), following a suggestion from Ilya (@iSazonov) in #14692 (comment):

^{The following is adapted from #14747 (comment), which contains some additional information about native argument-passing on Windows.}

PR #14692 introduces experimental feature PSNativeCommandArgumentPassing that will address parameter-passing woes when calling native programs with respect to embedded quoting and empty-string arguments, taking advantage of System.Diagnostics.ProcessStartInfo.ArgumentList, which:

on Unix-like platforms: fully solves all problems.
on Windows: solves the problem only for those programs that adhere to the quoting and escaping conventions used by Microsoft's C/C++ runtime.

While this is a great step in the right direction, it leaves out many Windows CLIs that do not play by these rules:

The most prominent exception is cmd.exe - and therefore calls to batch files: they accept only "" as an escaped ", not the \" required by the C/C++ convention); while Microsoft compiler-generated executables also support "", there are third-party programs that support only \")
An additional problem is that batch files unfortunately and inappropriately parse their arguments as if they had been passed from inside cmd.exe, which causes something like .\foo.cmd http://example.org?foo&bar to break due to & being misinterpreted as a statement separator. Using "http://example.org?foo&bar", i.e. quoting from PowerShell doesn't help, because PowerShell - justifiably - omits the quotes when it rebuilds the process command line behind the scenes, given that value contains neither spaces nor embedded " chars.
- This is especially problematic given that the CLIs of many high-profile environments (e.g., az.cmd for Azure, and the wrapper batch files that npm (Node.js's package manager) creates for (Java)script-based utilities that come with packages) use batch files as their CLI entry points, so that something like
- az ... 'http://example.org?foo&bar' predictably fails.
Calling cmd.exe /c "<command-line>" or cmd.exe /k "<command-line>" directly with a single-argument command line to be executed through a happy accident actually currently works as intended, without a workaround - and that behavior must be retained.
Many programs are particular about partial quoting of arguments, notably msiexec.exe with property arguments such as PROP="VALUE WITH SPACES"; purely syntactically, "PROP=VALUE WITH SPACES" (which is what PowerShell currently sends) should be equivalent (and if you let the C/C++ runtime / CLR parse it, is - the resulting verbatim string is PROP=VALUE WITH SPACES in both cases), but in practice it is not.
- PowerShell should not pay attention to the original quoting on the PowerShell command line in an attempt to emulate it when re-encoding behind the scenes; no such quoting may be present to begin with (e.g., PROP=$someValuePossiblyWithSpaces), and users generally shouldn't have to worry about such intricacies - see below.
Finally, calls to the WSH (Windows Script Host) CLIs cscript.exe (console) and wscript.exe - either directly or via associated script file types, notably .vbs (VBScript) and .js (JScript), behave poorly with \"-escaped embedded " characters; while the problem cannot be fully solved, ""-escaping results in better behavior: see below for details.

It's impossible for PowerShell to fully solve these problems, but it makes sense to make accommodations for these exceptions, as long as they are based on general rules (rather than individual exceptions) that are easy to conceptualize and document.

I believe it is vital to make these accommodations as part of the PSNativeCommandArgumentPassing experimental feature implemented in PR #14692 in order to solve the vast majority of quoting headaches once and for all.
They are detailed below.

For the remaining, edge cases there is:

--% for console applications, or, preferably, because it has fewer limitations and enables use of PowerShell variable values and expressions via string interpolation, cmd /c "<cmd.exe command line>".
Start-Process for GUI-subsystem applications with a CLI such as msiexec, which allows you to fully control the process command line by passing a single string to -ArgumentList (in a pinch you can also use it with console applications, but you lose stream integration).

Proposed technical implementation details

After PowerShell's own parsing, once the array of verbatim arguments - stripped of $nulls - to pass on is available:

On Unix-like platform:
- Pass that array to .ArgumentList - that is all that is ever needed.
On Windows:
- Except for the cases detailed below, also pass that array to .ArgumentList - behind the scenes; .NET then performs the necessary re-encoding based on the C/C++ conventions for us, and any conventional CLI should interpret the result correctly.
- The following exceptions may apply independently or in combination, and they require manual re-encoding by PowerShell (with assignment to .Arguments, as currently):
  - A current behavior that must be retained - i.e. no escaping of embedded " must be performed - is the very specific case of cmd.exe being called directly, with either the /c or the /k option followed by a single argument (with spaces) representing a cmd.exe command line in full.
    - See here for details, including the proposal for an optional additional accommodation that would make sense, namely to robustly support passing the command line following /c or /k as multiple arguments, by transforming it into a single-argument, double-quoted-overall form.
    - Implementation-wise, we'd get this additional accommodation almost for free, because we need it behind the scenes for calling batch files anyway, so as to support reliable exit-code reporting - see next point.
  - If the target command is a batch file:
    - use "" (rather than \") to escape embedded verbatim " (and ensure enclosure in syntactic "...", even if the value has no spaces)
    - "..."-enclose any argument that contain no spaces (such arguments are normally not quoted) but contain any of the following cmd.exe metacharacters: & | < > ^ , ; (while , and ; have no impact on arguments pass-through with %*, they serve as argument separators in intra-batch file argument parsing; this also applies to =, but, unfortunately, passing something like FOO=bar as "FOO=bar" conflicts with the accommodation for msiexec-style CLIs below).
    - Additionally, for reliable exit-code reporting, call the batch file via cmd /c "<batch-file> ... & exit" rather than directly; see below for the detailed rationale.
      - This means:
        
        Make cmd.exe the executable.
        
        Pass a single argument string /c "<batch-file> ... & exit", where <batch-file> path may need to be double-quoted and ... represents the space-joined list of the arguments quoted based on the rules above. Again, no escaping of any " characters ending up in the overall "..." string passed to /c need or must be performed.
        
        Note that on Windows versions before Windows 10, such a call fails if the batch-file path itself must be double-quoted (typically, due to containing spaces); the workaround is to determine the short (8.3) form of that path - which by definition doesn't require double-quoting - and use that instead.
  - Irrespective of the target executable, if any of the arguments have the form of a misexec-style partial-quoting argument, apply double-quoting only to the "value" part (the part after the separator):
    - Specifically, if an argument (a) matches regex ^([/-]\w+[=:]|\w+=)(.+)$, and (b) the part after = or : requires double-quoting (either due to containing spaces and/or an embedded " and/or, in the case of a batch file containing cmd.exe metacharacters), leave the part up to and including = or : unquoted, and double-quote only the remaining part.
    - Examples:
      - The following PowerShell arguments:
        
        FOO='bar none', -foo:$value (with $value containing verbatim bar none), /foo:bar` none (and even quoted-in-full variants 'FOO=bar none', ....)
      - would end up in the .Arguments command line as follows:
        
        FOO="bar none", -foo:"bar none", /foo:"bar none"
  - If the target executable is a WSH CLI - cscript.exe or wscript.exe - or the filename extension is one of the following WSH-associated extensions listed by default in $env:PATHEXT (which makes them directly executable): .vbs .vbe .js .jse .wsf .wsh, use "" rather than \" to escape " characters embedded in arguments; again see below for details.

Again, these are reasonable accommodations to make, which:

allow users to focus solely on PowerShell's syntax
should make the vast majority of calls just work.
are easy to conceptualize and document - the proposed tracing should help too.

I invite everyone to scrutinize these accommodations to see if they're complete, overzealous, ...

This is a chance to finally cure all native quoting / argument-passing headaches - even if only by opt-in.

(To experiment with the proposed behaviors up front (based on my personal implementation that sits on top of the current behavior), you can use Install-Module Native and prepend ie to command lines; if the proposed changes are implemented, such a stopgap will no longer be necessary, although it can still help on earlier versions.)

James Truher (@JamesWTruher)

I'm delighted that James Truher (@JamesWTruher) implemented #14692!

I hope we will not stop half way and finally close this story by fully implementing the proposal.

As it turns out, there's another accommodation worth making for batch files with respect to exit codes (now implemented in v1.3.1 of the Native module):

Direct invocation of batch files doesn't report their exit code, as reflected in $LASTEXITCODE, reliably; e.g.:

# Create a (temporary) batch file that provokes an error with an unsupported whoami.exe option, 
# and exits with `exit /b` *without an explicit argument* with the intent to *pass whoami.exe's exit code through*.
'@echo off & whoami -nosuch 2>NUL || exit /b' | Set-Content test.cmd

# Invoke the batch file and report the exit code.
PS> .\test.cmd; $LASTEXITCODE
0   # !! whoami.exe's exit code, 1, was NOT passed through.

The workaround is to use the following invocation in lieu of .\test.cmd: cmd /c ".\test.cmd & exit", as detailed in this Stack Overflow post.

Note that inside a cmd.exe session, %ERRORLEVEL% is set to 1 after calling the batch file.

To be clear: The problem lies with cmd.exe, but it is yet another accommodation we can make to improve the robustness of native-program calls.

The following:

fleshes out the proposal above with concrete examples
contains a list of bugs as of PowerShell Core 7.2.0-preview.5

`PSNativeCommandArgumentPassing`: Missing-accommodations examples, current bugs:

Setup:

Install-Module Native

Note:

The diagnostic output below, via argument-/command-line diagnosing function dbea (which uses -ie to use the ie function behind the scenes, which ensures the desired behavior); these functions are from the Native module. It is my hope that the need for ie will go away with proper implementation of PSNativeCommandArgumentPassing (though it will still be useful in older PS versions).
- First shows the verbatim arguments that the target batch file / conventional executable ends up seeing after parsing its own command line, i.e. the effective values.
- Then shows the raw command line (sans executable) that was passed (which by definition only applies on Windows): this is what PowerShell must construct from the given arguments and assign to System.Diagnostics.ProcessStartInfo.Arguments.

Accommodation A: Calling a batch file: Must escape embedded " as "":

PS> dbea -ie -UseBatchFile -- 'Andre "The Hawk" Dawson' 'another argument'
2 argument(s) received (enclosed in «...» for delineation):

  «"Andre ""The Hawk"" Dawson"»
  «"another argument"»

Command line (without executable; the value of %*):

  "Andre ""The Hawk"" Dawson" "another argument"

Note how the verbatim embedded "The Hawk" was escaped as ""The Hawk"", to ensure that the batch file properly recognized the entire token as a single argument.

Accommodation B: Calling a batch file: Space-less arguments (ones that normally do not trigger enclosure in "...") must still be double-quoted if any of the following cmd.exe metacharacters is present: & | < > ^ , ;

PS> dbea -ie -UseBatchFile -- 'http://example.org?foo=1&bar=2' 'another argument'
2 argument(s) received (enclosed in «...» for delineation):

  «"http://example.org?foo=1&bar=2"»
  «"another argument"»

Command line (without executable; the value of %*):

  "http://example.org?foo=1&bar=2" "another argument"

Note how the URL was double-quoted, despite not containing spaces.
Currently, this fails, both with and without PSNativeCommandArgumentPassing in effect, because in the absence of double-quoting cmd.exe (which parses the batch file's command line) interprets the unquoted & as a statement separator.

Accommodation C: Passing a PROPERTY=VALUE-style argument (and variations thereof, prefixed with / or - optionally combined with : instead of =) to any executable: Must selectively "..."-enclose only the VALUE part, if it contains spaces so as to satisfy the particular, nonstandard quoting requirements of executables such as msiexec (to conventional executables, this quoting variation makes no difference, so it is safe to use):

PS> dbea -ie -- INSTALLDIR=$PSHOME  'another argument'
2 argument(s) received (enclosed in «...» for delineation):

  «INSTALLDIR=C:\Program Files\PowerShell\7»
  «another argument»

Command line (without executable):

  INSTALLDIR="C:\Program Files\PowerShell\7" "another argument"

Note how INSTALLDIR=$PSHOME turned into selectively double-quoted INSTALLDIR="C:\Program Files\PowerShell\7"

Note:

Accommodation B and C must situationally be combined.
Accommodation B, due to the need for Accommodation C, must actually be performed in the context of a cmd /c "<batch-file> ... & exit" call - see below.

Accommodation D: Support reliable exit-code reporting from batch files:

PS> '@echo off & whoami -nosuch 2>NUL || exit /b' | Set-Content test.cmd; ie ./test.cmd; $LASTEXITCODE
1

Without ie, this currently reports 0 (both with and without PSNativeCommandArgumentPassing in effect), i.e. whoami's failure exit code was not reported.

For this to work as intended, PowerShell must translate ./test.cmd ... into cmd /c "c:\path\to\test.cmd ... & exit, as explained in detail here. Due to cmd.exe not expecting escaping inside the overall "..." string, the encoding of the arguments is the same as would be needed for Accommodation B in isolation.

Accommodation E: Use ""-escaping for WSH calls (direct and indirect calls of cscript.exe / wscript.exe) to result in less broken behavior:

PS> dbea -ie -UseWSH 'Andre "The Hawk" Dawson' 'another argument'
2 argument(s) received:

  «Andre The Hawk Dawson»
  «another argument»

Note how the embedded " were stripped - which cannot be avoided, unfortunately - but the argument boundaries were preserved, whereas applying the default \" escaping would result both in broken argument boundaries and literally retained \ chars - see below for details.

Accommodation F: Transform multi-argument direct cmd /c / cmd /k executable calls into single-argument, overall "..."-enclosed ones behind the scenes.

This accommodation is arguably the least important, but I think it would be beneficial, because it's one less thing for users to worry about; plus, the important Accommodation D would give us the implementation for free:

PS> ie cmd /c "C:\Program Files\PowerShell\7\pwsh" -noprofile -c " 'hi there' "
hi there

Without ie, this command breaks, both with and without PSNativeCommandArgumentPassing in effect, due to cmd.exe's own limitations.

If PowerShell transformed the above command line into the following, verbatim single-argument cmd /c form in .Arguments (which is what ie does), the command would work:

" "C:\Program Files\PowerShell\7\pwsh" -noprofile -c " 'hi there' " "

To simulate this with --%:

PS> & { $PSNativeCommandArgumentPassing='Legacy'
      cmd --% /c " "C:\Program Files\PowerShell\7\pwsh" -noprofile -c " 'hi there' " "
    }

For details, see #14747 (comment)

Fix 1: Calling cmd.exe /c / cmd.exe /k directly with a command line (as a single argument) must continue to work properly, as it (by lucky accident) always has, even with PSNativeCommandArgumentPassing in effect:

# Legacy behavior is actually *ok* in this case:
PS> & { $PSNativeCommandArgumentPassing='Legacy'; cmd /c ' echo "Andre ""The Hawk"" Dawson" ' }
"Andre ""The Hawk"" Dawson"

With PSNativeCommandArgumentPassing in effect ($PSNativeCommandArgumentPassing='Standard'), you'll now get (broken)
\"Andre \"\"The Hawk\"\" Dawson\" .

Such cmd /c calls have historically been valuable for working around quoting headaches in PowerShell.
They will continue to be useful:

For - hopefully very rare - quoting edge cases that the accommodations do not cover, as a superior alternative to --%, given that embedding PowerShell variable values and expressions is then easily possible via string interpolation.
Probably more frequently, to work around the PowerShell limitation of not providing raw byte handling in the pipeline, and therefore needing to resort to cmd.exe's |.

See #15239.

Fix 2: --% must continue to work even with PSNativeCommandArgumentPassing in effect:

--%, the stop-parsing symbol, is a per-call syntax override, and its functioning should not be affected by whether or not PSNativeCommandArgumentPassing in effect. While --% is virtually useless on Unix, on Windows there will still be - hopefully very rare - edge cases that require it, and there's also no need to break existing calls using it.

# --% use: OK only in legacy mode:
PS> & { $PSNativeCommandArgumentPassing='Legacy'
             cmd /c --% echo "Andre ""The Hawk"" Dawson"
    }
"Andre ""The Hawk"" Dawson"

With PSNativeCommandArgumentPassing in effect ($PSNativeCommandArgumentPassing='Standard'), you'll now get
(broken) "\"Andre" "\"\"The" "Hawk\"\"" "Dawson\""

See #15261.

Fix 3: Passing arguments such as -F: (looks like a PowerShell parameter, ends in :) must continue to work even with PSNativeCommandArgumentPassing in effect:

The following fails as of PowerShell Core 7.2.0-preview.5, because the -F: and foo arguments are unexpectedly merged behind the scenes:

PS> cmd /c echo -F: foo # On Unix, use: /bin/echo -F: foo
-F:foo  # !! Note the missing space.

See #15276

Fix 4: PSNativeCommandArgumentPassing must not break invocation of WSH scripts (VBScripts, JScripts) with arguments.

See #15289

Steve Lee (@SteveL-MSFT)

Steve Lee (@SteveL-MSFT), on a meta note similar to the one in #14025 (comment):

Two separate requests to discuss the issue at hand in the April community call were made (one by Ilya (@iSazonov), referring here directly, and by Justin Grote (@JustinGrote), referring here indirectly, via #1995 (comment)).

In the community call you glossed over these requests by saying that James Truher (@JamesWTruher)'s presentation had already covered the topic by his presentation on the new, experimental PSNativeCommandArgumentPassing feature.

This despite the fact that the very point of the issue at hand is to point out problems with the feature as currently implemented - and that is what needed to be discussed.

That the discussion was simply brushed aside again does not instill confidence in the concerns of the community being taken seriously.

On Windows, there is basic inconsistency between the way a batch file arguments and Windows executable arguments are parsed. with a batch file, the = seems to be a token separator, but not with the compiled executable. This was most surprising. Here's the transcript (echoit.exe is a very simple native app). All of the following is done from cmd.exe"

C:\Users\james>.\echoit.exe -foo=bar -foo="bar baz" "-foo=bar baz"
Argument 1 <-foo=bar>
Argument 2 <-foo=bar baz>
Argument 3 <-foo=bar baz>

C:\Users\james>type echoit.cmd
@echo off
echo %0
echo %1
echo %2
echo %3
echo %4
echo %5
echo %6
echo %7
echo %8
echo %9

C:\Users\james>.\echoit.cmd -foo=bar -foo="bar baz" "-foo=bar baz"
.\echoit.cmd
-foo
bar
-foo
"bar baz"
"-foo=bar baz"
ECHO is off.
ECHO is off.
ECHO is off.
ECHO is off.

from PowerShell, the experience is as follows:

PS>.\echoit.cmd -foo=bar -foo="bar baz" "-foo=bar baz"
"C:\Users\james\echoit.cmd"
-foo
bar
"-foo=bar baz"
"-foo=bar baz"
ECHO is off.
ECHO is off.
ECHO is off.
ECHO is off.
ECHO is off.
PS> .\echoit.exe -foo=bar -foo="bar baz" "-foo=bar baz"
Argument 1 <-foo=bar>
Argument 2 <-foo=bar baz>
Argument 3 <-foo=bar baz>

I didn't try a VB script.
My concern with this request would be to have PowerShell have to know too much about what is being executed. Right now there is no disambiguation about what is being executed and to change the way things are parsed when running a batch file vs an executable seems very improper to me. With regard to MSIEXEC.EXE specifically, i believe we will handle that just fine. If there is an issue with regard to an .exe, i really need to see the specifics.

James Truher (@JamesWTruher)

Thanks for engaging, James Truher (@JamesWTruher).
Before we get into specifics:

change the way things are parsed when running a batch file vs an executable seems very improper to me

It's not improper. It's an invaluable accommodation that relieves users of (most of) the burden of having to account for the anarchy of argument-passing on Windows.

Ensuring that arguments are ultimately passed as-is - based on PowerShell's parsing rules alone - to external programs as-is is a core duty of a shell.

While the limitations of Windows prevent a fully robust solution, we can provide a solution that covers most use cases, based on straightforward rules - and the ones laid out for the class of batch files (as opposed to hard-coding exceptions for specific executables) achieve that.

With regard to MSIEXEC.EXE specifically, i believe we will handle that just fine

No. passing something like foo='bar none' results in "foo=bar none" on the behind-the-scenes command line, which msiexec.exe (and others, such as msdeploy.exe) do not recognize.
Again, the proposed accommodation is based on an arguments matching a (regex) pattern, not on specific executables.

I didn't try a VB script.

Good point - I hadn't considered VBScript - I'll investigate.

James Truher (@JamesWTruher)

Quick update, James Truher (@JamesWTruher): PSNativeCommandArgumentPassing breaks invocation of VBScripts with arguments: see #15289

As for VBScript (WSH):

VBScript's command-line argument parsing is provided by the WSH (Windows Script Host) CLIs, cscript.exe (console) and wscript.exe (GUI), so with direct invocation of script files the same behavior applies to the following filename extensions listed in $env:PATHEXT, which includes both VBScript and JScript scripts and their variants, as well as WSH wrapper files:

.vbs .vbe .js .jse .wsf .wsh

WSH supports neither \"- nor ""-escaping of embedded "", but using "" results in less broken behavior, and is therefore another accommodation worth making - I've updated the initial post and the comment with the examples accordingly:

WSH parses a command-line token such as "Andre ""The Hawk"" Dawson" as 3 directly adjacent "..." substrings, which it implicitly joins to form single, verbatim argument Andre The Hawk Dawson. While this is broken in that the embedded " were stripped, at least the token was still recognized as a single argument.

By contrast, "Andre \"The Hawk\" Dawson" - which is what PSNativeCommandArgumentPassing currently does (as of this writing you must invoke scripts explicitly with cscript.exe, due to bug #15289) - is parsed as two arguments, and the \ characters are _retained: That is, "Andre \"The becomes the the first argument, as verbatim Andre \The, and Hawk\" Dawson" becomes the second, as verbatim Hawk\ Dawson.

To demonstrate the difference, assuming the following test.vbs script:

' Save with Windows-1252 encoding for the guillemets («») to render properly.
Wscript.Echo CStr(WScript.Arguments.Count) + " argument(s) received:" + vbLf

i = 0
for each arg in WScript.Arguments
  i = i + 1
  WScript.Echo "  «" + arg + "»"
next

WScript.Echo

Current PSNativeCommandArgumentPassing behavior: \"-escaping:

PS> cscript.exe .\test.vbs 'Andre "The Hawk" Dawson' 'another argument'
3 argument(s) received:

  «Andre \The»
  «Hawk\ Dawson»
  «another argument»

Note the broken argument partitioning and the literally retained \ chars.

Behavior with the proposed accommodation (""-"escaping"; you need Install-Module Native v1.4.1 or higher for the ie function to show this behavior):

PS> ie cscript.exe .\test.vbs 'Andre "The Hawk" Dawson' 'another argument'
2 argument(s) received:

  «Andre The Hawk Dawson»
  «another argument»

While the behavior is partially broken - what were meant to be embedded " were stripped - at least the argument was still recognized as a single argument and no characters were added.

James Truher (@JamesWTruher)

James Truher (@JamesWTruher), with respect to batch files:

with a batch file, the = seems to be a token separator

Specifically, unquoted <space> , ; = interchangeably act as argument separators in batch-file command lines (though in practice, it seems, it is only ever spaces that are used).

PS>  dbea -UseBatchFile 'a b', 'a,b', 'a;b', 'a=b'
7 argument(s) received (enclosed in «...» for delineation):

  «"a b"»
  «a»
  «b»
  «a»
  «b»
  «a»
  «b»

Command line (without executable; the value of %*):

  "a b" a,b a;b a=b

Note how PowerShell implicitly double-quoted 'a b', due to containing a space, but the other - spaceless ones - were passed unquoted - causing unexpected argument separation. That is, what you clearly meant to pass as verbatim a,b, for instance - on the PowerShell side single-quoting was used, after all - should also be received by the batch file as such - that is the job of a shell.

With the proposed Accommodation B (see above), even such space-less arguments will be double-quoted in batch-file calls, to ensure that with the exception of =, and the addition of & | < > ^:

# Call via the `ie` function, to activate the accommodations:
PS> dbea -ie -UseBatchFile 'a b', 'a,b', 'a;b', 'a=b' 'a&b' 'a|b' 'a<b', 'a>b' 'a^b'
10 argument(s) received (enclosed in «...» for delineation):

  «"a b"»
  «"a,b"»
  «"a;b"»
  «a»
  «b»
  «"a&b"»
  «"a|b"»
  «"a<b"»
  «"a>b"»
  «"a^b"»

Command line (without executable; the value of %*):

  "a b" "a,b" "a;b" a=b "a&b" "a|b" "a<b" "a>b" "a^b"

Note that arguments 'a&b' 'a|b' 'a<b', 'a>b' 'a^b' simply break the call with the current PSNativeCommandArgumentPassing implementation, because the unquoted & | < > ^ are then interpreted as cmd.exe metacharacters.

The reason for the = exception is that it conflicts with Accommodation C for msiexec-style executables (where the a= part of a=b must be unquoted), and I think the latter should be given precedence in this case.

Fortunately, passing a=b through as a single argument as part of relaying the command line via %* isn't affected, as you can see from the output above, and over time I expect more and more batch files to be nothing more than CLI entry points with pass-through arguments (e.g., for Python CLIs, such as the az.cmd batch file that is the Azure CLI's entry point).

The initial post details the proposed accommodations and their benefits and this follow-up comment provides examples of what we would gain.

Perhaps it is helpful to complement that with examples of what won't work, unless these accommodations are implemented:

First, a quick aside: Given the number of bugs summarized above that have surfaced in only a short period after release of PowerShell Core 7.2.0-preview.5, it is evident that more comprehensive tests are needed.

The examples show the current behavior with PSNativeCommandArgumentPassing in effect, as of PowerShell Core 7.2.0-preview.5; their names correspond to the accommodation examples above; all commands can be made to work properly with Install-Module Native and by prepending each invocation with ie (e.g. ie .\temp.cmd ...)

Unsupported Scenario A: Inability to pass arguments with embedded " to batch files:

"@echo off`necho [%1]`n" > temp.cmd; .\temp.cmd 'Luke "Aches & Pains" Appling'; Remove-Item temp.cmd
["Luke \"Aches]
The system cannot find the path specified.

As you can see, not only was the argument not recognized as a whole, the command broke, because - due to the embedded " being escaped as \" instead of "" - the from cmd.exe's perspective effectively unquoted & was interpreted as a statement separator.

Unsupported Scenario B: Inability to pass space-less arguments that contain & , ; ^ | < > to batch files:

Using Azure's az CLI, which is implemented as wrapper batch file az.cmd that calls a Python script, as an example:

# Note the '&count=10' part of the URL
PS> az.cmd rest --method get --url 'https://example.org/resources?api-version=2019-07-01&count=10'
<Azure-specific error message>
'count' is not recognized as an internal or external command,
operable program or batch file.

Since the URL by definition contains no spaces, PowerShell passes it unquoted, so that the unquoted & is again interpreted as cmd.exe's _statement separator.

In other words: with direct invocation, it is impossible to pass a URL that contains & character to a batch file.

Unsupported Scenario C: Inability to pass PROPERTY="VALUE WITH SPACES" style arguments to msiexec-style executables:

# FAILS due to invalid syntax.
# The fact that the CLI help dialog pops up implies that.
# (A syntactically correct call would result in a quiet no-op, due to use of /quiet)
PS> $dir='c:\program files\foo'; msiexec /quiet /i foo.msi INSTALLDIR=$dir

Because PowerShell passes "INSTALLDIR=c:\program files\foo" rather than the INSTALLDIR="c:\program files\foo" required by msiexec - quoting of the value part only - msiexec encounters a syntax error.

Unsupported Scenario D: Inability to reliably report a batch file's exit code:

PS> '@echo off & whoami -nosuch 2>NUL || exit /b' | Set-Content test.cmd; .\test.cmd; $LASTEXITCODE
0

That is, even though whoami reported exit code 1 and the batch file was exited as result of that, PowerShell saw exit 0, mistakenly implying that the batch file succeeded.

Unsupported Scenario E: Inability to call WSH (VBScript, JScript) scripts with arguments with embedded " while preserving at least argument boundaries:

# To work around #15289, cscript.exe is explicitly used for invocation, but the behavior would be the same without it.
PS> ' WScript.Echo WScript.Arguments(0)' > temp.vbs; cscript .\temp.vbs 'Luke "Aches & Pains" Appling'; Remove-Item temp.vbs
Luke \Aches

Note how the argument wasn't recognized as a whole, and a \ was inserted, resulting from the \"-escaping that is applied.
As detailed above, there is no complete solution here, but using ""-escaping at least preserves argument boundaries (while stripping embedded ").

Unsupported Scenario F: Inability to call cmd.exe with /c or /k with individual arguments if both the executable path and one of the arguments contain ":

PS> cmd /c 'C:\Program Files\PowerShell\7\pwsh' -noprofile -c "'hi there'"
'C:\Program' is not recognized as an internal or external command, operable program or batch file.

I don't know if the behavior of scenario F should be changed. This is simply the behavior of cmd.exe. To make it work just use

cmd.exe /c@ 'C:\Program Files\PowerShell\7-preview\pwsh' -noprofile -c "'hi there'"

This form of the command works in Windows Powershell 5.1, it works in the current powershell preview with PSNativeCommandArgumentPassing enabled and it works in cygwin bash and in ubuntu wsl bash.

I'm just not a huge fan of detecting a specific exe file and behaving differently. Detecting a specific file type (.vbs or .bat) would also not be ideal, but would be IMHO much cleaner then detecting a specific exe.

Kirk Munro (@KirkMunro)

this is based on internal partner feedback using this experimental feature

You're confirming Kirk Munro (@KirkMunro)'s point.

The sad thing about this decision is that in this case there is no need to choose between serving favored groups and quality-of-life improvements for the community at large.

Introducing another setting will just create more confusion while providing no benefit at all; on the contrary.

Only two settings are needed:

Legacy - keep everything as is, for those who need to write code targeting both editions on Windows (and possibly older PS Core versions on Unix). The Byzantine workarounds required to work around the broken legacy behavior in that case are then unavoidable.
Standard - make PowerShell act like a proper shell as much as possible on Windows, for those free to target v7.2+ only, which requires implementing the proposed accommodations . By contrast, with the (lucky) exception of cmd /c, the legacy behavior is hopelessly broken, so introducing a selective opt-back-in setting is both confusing and utterly unhelpful.

What you call "magic" is the very opposite from the user's perspective:

It enables users to focus on PowerShell's syntax alone, trusting the shell to pass arguments on as specified solely by its own rules - that is a core mandate of a shell. That the act of passing arguments to external programs requires additional work behind the scenes is (a) unfortunate historical baggage on Windows that cannot be avoided and (b) should be an implementation detail that the user is shielded from.

By not providing a single setting that combines .ArgumentList with the selective Windows accommodations proposed, you're forever saddling Windows users with quoting headaches and obscure failures.
As stated, batch files as CLI entry points are common (Azure, npm-installed script-based executables, ...) and users may not even be aware that they're calling batch files, but - more importantly - they shouldn't have to be.

In the absence of a proper solution, the ie function from the Native module (Install-Module Native) can be used: it incorporates all proposed accommodations, and equally works in Windows PowerShell v3+ as well as in all PowerShell Core versions, irrespective of the availability of the PSNativeCommandArgumentPassing feature and its setting; just prepend ie to your native-executable calls.
Perhaps needless to say, needing a third-party module to make PowerShell fulfill one of its core mandates as a shell is a sad state of affairs.

As part of my ongoing withdrawal (in both senses of the word): I've said all that I have to say, and I'm unsubscribing from this thread.

README.md upated

Michael Klement (@mklement0)

I don't quite get this. There was a lot of resistance to "fixing" argument passing by "special casing" this feature for certain executables. But in the end, the team opts to "special case" the feature for these executables anyway - except not to fix the issues but drop back to the old (legacy) parsing mode. That's a head scratcher.

This new approach feels more like a "Hybrid" mode where some executables get the old parsing treatment. Is that list configurable? What if we discover another native Windows app that doesn't behave well with Standard mode?

Seems like this feature should stay experimental until the issues with cmd.exe, msiexec and c/wscript can be worked out. BTW Michael Klement (@mklement0), thanks for the Native module. It's quite handy.

One last note, the recent loss of community support should be concerning to the team. Something is clearly not working right. Don't get me wrong, I don't think the team should merge community PRs willy / nilly. There needs to be a high quality bar to ensure new PowerShell releases are high quality and avoid breaking changes as much as possible. But Michael Klement (@mklement0) has made some in-depth and well researched suggestions here that appear to be falling on deaf ears. :-(

The saga continues at #15408 (comment)

James Truher (@JamesWTruher)

Pretty sure doing that was one of the main premises James Truher (@JamesWTruher) was working from. I'd agree that's probably going to cause more confusion than it helps ultimately, though. 🤔

There are two things I can think of which might inform some of this, but I yet haven't fully explored...

There is a /s modifer for cmd /c and cmd /k which deliberately alters the behaviour of quoted arguments, that you may also need to factor. The wording for exactly how that works in cmd /? is somewhat confusing, but I have historically found circumstances (particularly in batch files) where adding /s has made all the difference (although I'd have to rummage to find anything concrete).

I only call this out because, since you are reworking quoting for cmd /c "..." and cmd /k "...", you may also need to rework quoting for cmd /s /c "..." and cmd /s /c "..." to match that behaviour, too.
The documentation in help exit does not expressly call out the behaviour you describe in Implement accommodations for Windows CLIs, notably batch files and misexec-style programs as part of experimental feature PSNativeCommandArgumentPassing #15143 (comment), so is it a valid test? As I read it, EXIT [/b] [exitCode] makes both /b and exitCode optional, and only says that /b exits a batch script, not the interpreter (which was actually wrong up until approx. Vista). The call exit /b does not specify an exit code, only that a batch file should exit. There is no help-documented default exit code if exitCode is not passed (I've always assumed 0), nor does it expressly state that it passes through any prior errorlevel.

As such, in whoami -nosuch 2>NUL || exit /b, isn't that technically saying if whoami exits with non-zero, exit from the batch, not if whoami exits with non-zero, exit from the batch with the exit code of whoami? (the exit code being arbitrary/undefined, and I've always assumed as 0).

(I've always used IF NOT "{%ERRORLEVEL%}"=="{0}" EXIT /b %ERRORLEVEL% as my terminate batch with prior exit code approach, but maybe that's never been necessary.)

Other things I can think of, which may or may not be relevant...

if we're building a list of exceptional quoting rules, find.exe '"hello"' hello.txt has always been a pain. It's even worse if you want to interpolate: find.exe "`"hello $myvar`"" (even moreso if you want to put that in Markdown!)
also consider forfiles.exe, which has its own expectations for /c (including /c cmd /c "foo") and its own processing of 0xHH hex numbers -- forfiles.exe /c "cmd /c 0x22runme /x @fpath0x22".
there is also special handling of ^ which can act, in some cases, as an escape character for Windows batch and at least one of the Win32/ShellExecute command-line building args.

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.

Navigation Menu

Repository navigation

Implement accommodations for Windows CLIs, notably batch files and misexec-style programs as part of experimental feature PSNativeCommandArgumentPassing #15143

Summary of the new feature/enhancement

A Guide to this Proposal:

Proposed technical implementation details

`PSNativeCommandArgumentPassing`: Missing-accommodations examples, current bugs:

6 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Implement accommodations for Windows CLIs, notably batch files and misexec-style programs as part of experimental feature PSNativeCommandArgumentPassing #15143

Description

Summary of the new feature/enhancement

A Guide to this Proposal:

Proposed technical implementation details

Activity

iSazonov commented on Apr 2, 2021

mklement0 commented on Apr 8, 2021

mklement0 commented on Apr 19, 2021

PSNativeCommandArgumentPassing: Missing-accommodations examples, current bugs:

mklement0 commented on Apr 21, 2021

JamesWTruher commented on Apr 21, 2021

mklement0 commented on Apr 21, 2021

mklement0 commented on Apr 21, 2021

mklement0 commented on Apr 22, 2021

mklement0 commented on Apr 22, 2021

mklement0 commented on Apr 25, 2021

TSlivede commented on Apr 25, 2021

6 remaining items

mklement0 commented on May 6, 2021

rkeithhill commented on May 12, 2021

mklement0 commented on Jun 18, 2021

vexx32 commented on Jun 22, 2021

jimbobmcgee commented on Feb 16, 2022

microsoft-github-policy-service commented on Nov 23, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`PSNativeCommandArgumentPassing`: Missing-accommodations examples, current bugs: