Repository navigation
Make the stop-parsing symbol (--%) work on Unix #3733
Description
Related: #3734
For the stop-parsing symbol, --%, to work sensibly on Unix:
-
[preferably] either:
/bin/sh, the default shell, must be invoked, passing it a command line as a single string via-c.
As a - in my view beneficial - side effect,shwould perform globbing on unquoted tokens. -
or: PowerShell must still first interpret the tokens of the pass-through command line to strip syntactical quoting, in addition to expanding environment/shell variable references.
That may still run afoul of users' expectation that globbing should be performed.
Note: On Windows PowerShell can get away with only expanding %<name>%-style environment-variable references and then invoking the target utility without shell involvement, but that's not an option on Unix, where the shell is expected to perform tokenization up front and pass an array of literal arguments instead (with syntactical elements such as quoting characters removed).
To illustrate the problem with the current behavior (run the commands on a Unix platform):
> /bin/echo --% 'hi, mom'
'hi mom' # The enclosing single quotes were treated as *literals*Trying to pass an awk command that from sh looks like this: awk -F\" 'BEGIN { print "hi, mom" }'
> awk --% -F\" 'BEGIN { print "hi, mom" }'
/usr/bin/awk: syntax error at source line 1
context is
>>> ' <<<
/usr/bin/awk: bailing out at source line 1The array of arguments getting passed to awk breaks down as follows ($<n> represents the nth positional argument):
$1=[-F"]
$2=['BEGIN]
$3=[{]
$4=[print]
$5=[hi, mom]
$6=[}']
As you can see, the intended argument boundaries weren't respected, and the ' quoting characters were retained.
In fact, there's currently no way to make the above command work with --%, because no one is interpreting the quoting and removing the syntactical quoting characters.
Currently, only the following truly Byzantine invocation - not involving --% - can make the above awk command work:
/bin/sh -c "awk -F\\\`" 'BEGIN { print \`"hi, mom\`" }'"Environment data
PowerShell Core v6.0.0-alpha (v6.0.0-alpha.18) on macOS 10.12.4
PowerShell Core v6.0.0-alpha (v6.0.0-alpha.18) on Ubuntu 16.04.1 LTS
Activity
BrucePay commentedon May 17, 2017
--% is very specifically a Windows thing designed as a (rather poor) workaround for the inconsistencies in command line processing on Windows. I'm not sure it has or should have a role on non-Windows systems. It would be better if PowerShell quoting "just worked" on these systems.
mklement0 commentedon May 23, 2017
Bruce Payette (@BrucePay):
That's a commendable goal, but I fear it is unrealistic:
While automatically applying globbing on Unix platforms when invoking external utilities is helpful, I don't think you can ever make PowerShell-parsed command lines "just work" as POSIX-like shell command lines.
Take the following example, which relies on (POSIX-compliant) shell parameter expansion:
If you try this from PowerShell:
On a secondary note:
I presume that the
%in--%is a nod to thecmd.exesyntax of environment-variable references (%<name>%), so, I suppose, the POSIX-shell equivalent would be--$, but I'm not sure that platform distinction is worth introducing.5 remaining items
TSlivede commentedon May 24, 2017
But also on Unix
--%is currently the only reliable and portable way to pass"to a child command -- this will hopefully change soon.If
--%is changed, old scripts, for example ported from windows (or scripts, that need to run on multiple versions of powershell), will break, therefore I'd strongly prefer a different symbol than--%.TSlivede commentedon May 24, 2017
Yes, that's a problem, therefore I suggested the single line here-string (for example
@@' ... '@@or similar, must be well documented, maybe suggested in the documentation near to--%) -- a three line here-doc just looks a little bit strange, although I don't know if it's worth to add a new syntax...mklement0 commentedon May 24, 2017
No, unfortunately, it is broken on Unix, as my first post tried to demonstrate:
As you can see, the single quotes were preserved, which is not what you get when you execute the same command (sans
--%) in a POSIX-like shell:$ /bin/echo 'hi, mom' hi, momPassing parameters (arguments) works very differently on Unix, where it is the shell that provides the argument parsing, not the target command.
I haven't read through all the comments on the related issues and your RFC in full yet, but it's clear to me that there is no need to reconstruct a command-line string on Unix after PowerShell has done its own parsing - instead, pass the PowerShell-parsed/expanded arguments as-is to the underlying system function (e.g.,
execv()), as an array of literal tokens.It is only the anarchy that is Windows command-line handling that requires the - brittle - reconstruction of a single-string command line behind the scenes.
In POSIX-like shells on Unix, it is the shell that provides said preprocessing and then passes an array of as-is arguments - after having performed quote removal to remove the shell-relevant-only syntactical quotes.
I haven't considered all potential backward-compatibility issues, but if there weren't any, that's undoubtedly the way to go.
TSlivede commentedon May 24, 2017
I wouldn't consider
--%broken: It behaves equally to windows, and as the normal calling is broken (like you correctly noticed in #3734) some garbage likeis the only reliable way that works in all powershell versions on linux and windows (only v3 and above).
I completely agree, although I don't see a big problem with the construction of one argument string -- but it's just essential, that the commandline is splitt into the same strings it was previously constructed from. The current behavior is fatal. Ok, skipping construction of the single string and splitting afterwards would improve performance, but that's the only reason I see. Skipping this joining and splitting would require different code branches for linux and windows and require p/invoke on linux AFAIK.
TSlivede commentedon May 24, 2017
One more reason why
--%isn't broken: https://github.com/PowerShell/PowerShell-Docs/blob/staging/reference/6/About/about_Parsing.md states, thatIt does not state, how that literal line is split into multiple tokens on Unix -- And on windows it doesn't need to split the literal, as the child executable gets this as one string.
TSlivede commentedon May 24, 2017
Back to the topic -- Although I think
--%should keep working, I thought again about something like--$,that also stops parsing, but then passes the remaining line as argument to
bash -c ....Such a syntax would help bash users when they start using powershell, this way helping to aquire new users and therefore maybe actually worth the new syntax addition.
(Although I absolutely don't like the syntax of
--%and think it should never have been added the way it is.)mklement0 commentedon May 24, 2017
There is no disagreement except that I said
--%is broken on Unix - you may choose to call this inapplicable instead, which is fine (I think that was Bruce Payette (@BrucePay)'s point) - it was, after all, devised for Windows.And adapting it to work analogously on Unix was the whole point of creating this issue.
If the answer is: don't use this on Unix, that's fine - if so, it deserves documenting and possibly a warning at parsing / runtime.
I do see a big problem with that: Don't assume how the target program will interpret its arguments. The fact that different programs do so differently is one of the existing pain points on Windows - see this SO answer of mine.
That's perfectly appropriate, if feasible (I don't know enough to comment).
TSlivede commentedon May 24, 2017
I completely agree with both.
Yes, that's the pain on Windows. On Unix however, the commandline is always split by the .Net Core framework -- if the commandline is build compatible to the Unix .Net Core version of
Process.Start/ProcessStartInfo.Arguments, the arguments will always reach the child executable correctly.BTW - nice answer on SO
SteveL-MSFT commentedon Sep 6, 2017
@PowerShell/powershell-committee reviewed this. due to inherent differences between Windows and Unix on argument processing,
--%was originally designed for Windows and we don't want different behavior for this switch on Windows on Unix. We are looking at making changes to improve native command arg processing on both Windows and Unix so that--%isn't required (where possible)