Repository navigation
globbing for native commands is too agressive #3931
Description
Steps to reproduce
Intuitively globbing should not kick-in inside the single-quoted strings.
echo '11:1' | grep '.*:.'
Expected behavior
Works, output is 11:1, like in bash.
Actual behavior
Cannot find drive. A drive with the name '.*' does not exist.
At line:1 char:1
+ echo '11:1' | grep '.*:.'
+ ~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (.*:String) [], DriveNotFoundException
+ FullyQualifiedErrorId : DriveNotFound
The error is pretty confusing for a unix user.
Workaround
Escape * by a backtick in the regex.
Environment data
> $PSVersionTable
Name Value
---- -----
PSVersion 6.0.0-beta
PSEdition Core
BuildVersion 3.0.0.0
CLRVersion
GitCommitId v6.0.0-beta.2
OS Darwin 16.6.0 Darwin Kernel Version 16.6.0: Fri Apr 14 16:21:16 PDT 2017; root:xnu-3789....
Platform Unix
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
Activity
mklement0 commentedon Jun 5, 2017
Just to state it explicitly: globs shouldn't be expanded inside
"..."(double-quoted strings) either, which currently happens too:Only unquoted tokens should ever be subject to globbing, as in POSIX-like shells.
latkin commentedon Jun 5, 2017
Just noticed this after moving to Beta on OSX. Maybe a recent regression?
It makes using
curlimpossible if you have URL query parameters.vors commentedon Jun 5, 2017
Yes, not really a regression, it a new feature in beta-1
#3643
latkin commentedon Jun 6, 2017
Is there a way to disable globbing entirely and opt back in to the Windows-style behavior? Even when it works by design, I kind of hate it.
e.g.
git add *used to just work, but with globbing I'd need to dogit add '*'vors commentedon Jun 10, 2017
For this specific case
git add .may worklatkin commentedon Jul 12, 2017
Bump. Any update? This is preventing me from moving to beta on non-Windows.
vors commentedon Jul 12, 2017
Polite ping Bruce Payette (@BrucePay)
vors commentedon Jul 27, 2017
Another example of native utility that became unusable is
youtube-dl(or anything that takes url for that matter)rkeithhill commentedon Oct 11, 2017
And yet another example of globbbing messing up invocation of a native executable:
BTW the suggested workaround of escaping the
*doesn't work because`*is passed to the executable.rkeithhill commentedon Oct 11, 2017
So I was trying out my home grown echoargs in my WSL/Bash shell and noticed that given this command (in Bash):
You get this output:
RE globbing, how does Bash know to glob the first argument
g*but not the second oneFOO/*?mklement0 commentedon Oct 11, 2017
Bash (per POSIX):
only ever applies globbing to unquoted tokens;
*is subject to globbing,'*'and"*"are not (or a singly escaped*,\*)."foo"*would still match all files whose names start withfoo, because the*is unquoted.passes unquoted tokens that do not match any filesystem objects through as-is (in Bash, you can opt into different behavior, but that is not part of the POSIX standard); in your example,
FOO/*presumably didn't match anything and was therefore left untouched.My assumption has been that the idea behind PowerShell's native globbing is to emulate POSIX-shell rules (is that not true?), so in your earlier
conanexample, if you wanted to passPkgName/*through, you'd have to quote it to exempt it from globbing:4 remaining items
rkeithhill commentedon Oct 13, 2017
BTW a question for the Bash savvy. We expect that some folks will continue to live in Bash but we want them to be able to use PowerShell scripts in our repo. So we've shebang'd them and committed them with chmod=+x.
The problem comes when they want use Bash globbing with a PowerShell command. Bash globbing space separates the files and that messes up our PowerShell command. Internally, it can do the wildcard resolution but it is kind of sucky to have to tell Bash users they have to put wildcards in quotes when calling PowerShell scripts. Is there an option in Bash, to get it to generate glob lists that are comma separated?
mklement0 commentedon Oct 13, 2017
Lincoln Atkinson (@latkin):
No: the globbing is only applied when calling external utilities, on Unix, so nothing changes for calls to cmdlets / functions / *.ps1 PS scripts - or at all on Windows.
Note: Globbing is applied when passing arguments to an executable PowerShell script with a shebang line, as it technically is an external utility too.
mklement0 commentedon Oct 13, 2017
Keith Hill (@rkeithhill):
I don't think there is such an option - arguments are strictly space-separated in the Unix world, and there is no concept of an array-valued argument in the shell.
You could use a
ValueFromRemainingArgumentsparameter in your PowerShell scripts, but that limits you to 1 wildcard expression and notably precludes use of the parameter name in the invocation.As an aside: even a comma-separated list wouldn't help you, because PowerShell doesn't recognize arrays in arguments passed to it from the outside; e.g.,
1,2would be interpreted as scalar string1,2, and1, 2would be interpreted as 2 arguments:1,and2.rkeithhill commentedon Oct 13, 2017
So Bash users need to know to quote wildcard arguments. Guess that's just the way it'll have to be.
Well that appears to be a new bug. Has that been filed yet? So how do I direct my script users to pass array args to my PowerShell script from Bash? Sigh...
mklement0 commentedon Oct 13, 2017
Keith Hill (@rkeithhill):
I agree that passing arrays from the outside would be nice, but has that really ever worked?
I discovered the issue a while ago and assumed it was a by-design limitation of the CLI's argument parsing, similar to how all arguments are interpreted as literal strings.
rkeithhill commentedon Oct 13, 2017
Well, it is something that
PowerShellusers are use to e.g.:The question is what do Bash users expect? Presumably us PowerShell users will be using PowerShell. However, we will tell our Bash buddies how to run our PowerShell scripts and we're going to have to know that the array literal syntax we're used to in PowerShell isn't going to work in Bash.
mklement0 commentedon Oct 14, 2017
They expect to pass a list (array) - whose semantics are known to the target utility only - as either a single whitespace-less argument - e.g., to pass column names
pidandcommto utilitypsvia its-ooption:ps -o pid,commor as a quoted argument, if the value contains whitespace or other shell metacharacters - e.g.,
ps -o 'pid, comm'To Bash it is just a single argument in either case.
So, at least with how the PowerShell CLI currently works, the answer is again:
From within PowerShell, if you call an external utility - including a shebang-line PS script - with an array argument:
if what would normally be an array in PS has no embedded whitespace (e.g.,
1,2or'a','b'), it is NOT treated as an array and passed as a single argument.If the array elements are space-separated, they turn into individual arguments, by virtue of converting the array to a space-separated list of its elements (e.g.,
1, 2turns into1 2, seen by the target utility as separate arguments1and2.al-ign commentedon Oct 31, 2017
Oh, at last I found this issue.
There is same problem (at least as I see it) with invoking native commands with variables containing special symbols in options/arguments.
There is a real-life example when globbing interfere when it shouldn't - SELinux file context management through semanage (of course I leared this hard way, in the middle of writing deployment script).
I wrote this 'mini-test' to demonstrate it.
I have this behavior on PS 6.0.0-beta.9 on CentOS 7 1611
mklement0 commentedon Oct 31, 2017
I think at this point there is agreement that "Unix-native" globbing in PowerShell is broken, but we don't know yet how it will be fixed.
It's been laid out here how POSIX-like shells handle globbing, which decide whether to apply globbing based on the distinction between quoted and unquoted tokens - and anyone with Unix shell-scripting experience is aware of that.
Furthermore, even unquoted variable references are subject to globbing (e.g., in Bash:
var='*.txt'; echo $var # globbing happens, because $var is unquoted)Both concepts are alien to PowerShell, where
*.txtand'*.txt'are treated the same.$varand"$var"exists, but is entirely unrelated to globbing (and the need to pass a value with embedded whitespace as a single argument); it merely forces stringification.Two worlds collide here, and something's gotta give.
Adopting the quoted-vs.-unquoted distinction at least for literal unquoted tokens for calls to external utilities seems like a reasonable compromise to me,
but perhaps there's a different solution - we have yet to hear from the powers that be.(a fix is underway)As an aside re
--%:--%is not the answer not only because you then cannot use PS variables, but because it was designed for Windows and still behaves exclusively that way:Because it doesn't know It treats single quotes as syntactic elements, they become part of the argument to pass:
/bin/echo --% 'foo, bar'results in 2 arguments,'foo,andbar'- note the embedded single quotes.It will expand cmd-style environment variable references even on Unix (e.g.,
%HOME%), yet doesn't recognize Bash-style ones (e.g.,$HOME).A decision was made not to adapt
--%to Unix (whether as--%or with a distinct name) -see #3733 (comment)
iSazonov commentedon Oct 31, 2017
Michael Klement (@mklement0) Could you please review ##5188 ?
mklement0 commentedon Oct 31, 2017
Ilya (@iSazonov) Oops! Sorry I missed that a fix is already underway - will take a look.
Glob native command args only when not quoted
Glob native command args only when not quoted