Parsing of compound command-line tokens into arguments is surprising

Steve Lee (@SteveL-MSFT)

The primary purpose of this issue to highlight how the existing behavior may be surprising to newcomers (and even for seasoned users hard to remember and therefore predict in specific situations).
Given that the behavior has been around since the beginning, changing it is probably not an option, but documenting how the behavior may be surprising is worth doing; while some aspects of the behavior can be inferred from about_Parsing, the overall implications aren't clear.
- Steve Lee (@SteveL-MSFT): This issue subsumes Argument-mode parsing: Why is a subexpression at the start of an unquoted token treated differently than a variable reference? #3217, which therefore no longer needs the Documentation Needed label.
While this may ultimately become a documentation-repo issue, I would first like to solicit some feedback as to what code changes, if any, should be considered, after all (there are edge cases), and as to whether my summary is accurate.
While there may be good technical reasons for these behaviors, the problem is that they may be unexpected, especially given the deviation from how cmd.exe and bash handle these scenarios.
All surprising behaviors can be avoided with proper quoting - which is also worth pointing out in the docs - but the need to quote negates one of the convenient aspects of argument syntax. If at the end of the day you cannot remember whether a particular token requires quoting, your choices are to resort to trial and error, or to quote everything, which is cumbersome.

Note: By compound command-line token I mean a logically single token composed of the following syntax constructs by direct concatenation:

barewords (unquoted strings without special meaning; e.g., foo)
quoted strings (e.g.; 'bar', "baz")
variable references ($foo) and subexpressions ($(...))

Examples:

bar${foo} and ${foo}bar and $($foo)bar
foo"bar" and "bar"foo

The surprising aspects are:

Asymmetry: Whether a quoted string or subexpression comes first matters:

Placing the quoted string or subexpression first results in 2 arguments:

PS> Write-Output 'hi'there
hi
there

> Write-Output $($env:HOME)/Documents
/Users/jdoe
/Documents

Otherwise, a single argument is passed, as expected:

PS> Write-Output hi'there'
hithere

PS> Write-Output Documents:$($env:HOME)  # Note: just $env:HOME without $() is not affected.
Documents:/Users/jdoe

`.` following a leading quoted string is interpreted as [string] property access

PS> Write-Output 'hi'.there
   # NO output, because an attempt is made to access property .there on the string instance

`.` as the initial char. followed by a quoted string results in 2 arguments

PS> Write-Output .'hi'
.
hi

`-` as the initial char. in something that turns out not to be a parameter name results in 2 arguments with the presence of `:` or `.`

`-` as the initial char. causes embedded variable references not to be expanded

Environment data

Written as of:

PowerShell Core v6.0.2

Michael Klement (@mklement0)

Michael Klement (@mklement0) wrt your comment,

deviation from how cmd.exe and bash handle these scenarios.

Just so you'll have the context, traditional shells like bash are 'expand and parse shells'. They work as follows:

a line of text is read
quote removal is done, variables are expanded inline
The expanded line is then parsed and executed.
It is important to note step 2 - the values of variables are expanded in place as strings. This could not work for PowerShell with live object. Even if we went with some form of deserialized object in the string, the performance impact would be unacceptable.

PowerShell language processing is much more akin to python or ruby:

a line of text is read
the text is tokenized and parsed (there are deals between the tokenized and the parser so I made it one step.)
AST/code are generated and executed.

The expand-and-parse approach results in a particular set of characteristics. The parse-then-execute approach results in a different set of characteristics. PowerShell will never behave exactly like bash, It's fundamentally different. Our hope with PowerShell, was to have an equivalently flexible interactive experience while retaining the ability to script with true objects.

Whether a quoted string or subexpression comes first matters:

In both cases, you simply have a delimited token. '...' or $(...). After the trailing delimiter, scanning for the next token begins. Spaces are not needed to separated the tokens. The only thing that matter is that a complete token has been read.

following a leading quoted string is interpreted as [string] property access

Not exactly. If a '.' appears after a completed argument (string literal, variable, etc) then it will be treated as a member accessor expression, with the constraint that there can be no space before the '.' and the preceding token.

PS[1] (130) > write-output @{a=1;b=2}.count # Literal hashtable
2
PS[1] (131) > write-output "abc".length  # Literal string
3
PS[1] (132) > write-output (get-date).dayofweek # Cmdlet call
Thursday

as the initial char. in something that turns out not to be a parameter name results in 2 arguments with the presence of ':'

No. The type of a token is determined at parse time and only at parse time. How a parameter token (-param:) gets treated eventually depends on the type of command that is being bound. With [cmdletbinding()], if there isn't a format parameter corresponding to the actual parameter, you get an error. With simple functions, if there is a formal parameter with a matching name, then it gets bound otherwise it gets added to $args. If it's a native command, it gets passed as a string since that's all that can be passed to a native command. There is a fiddle that tries to stitch the two tokens "-parm:" and "value" back into a single string for literal arguments and there are requests to extend this fiddle. I'm beginning to think that would be a bad idea since the current hack seems to be causing people to make untrue assumptions. The more special cases there are the harder it is to explain.

Bruce Payette (@BrucePay)

Bruce Payette (@BrucePay):

First things first:

I appreciate the detailed response and illuminating peek behind the curtain.

Any changes to the existing behavior are probably too risky, but I want us to have clarity what a better approach might look like if and when the time comes where a change is feasible.

In the meantime, the list I've compiled is hopefully useful to document for others coming from a Bash / cmd background how the actual behavior may differ from their expectations.

PowerShell will never behave exactly like bash, It's fundamentally different.
the values of variables are expanded in place as strings. This could not work for PowerShell with live object.

Absolutely, in general, but in the case at hand - the compound tokens I describe - we are also dealing with strings - at least that's the only sensible interpretation.

One way to simplify the behavior is to consistently treat the components of a compound token as separate arguments, which is a simple rule that would be easy to remember.

All you would then have to tell Bash / cmd users is: in order to join distinct syntax constructs to form a single (string) argument, always use string interpolation explicitly, i.e., enclose the entire token in "...".

In both cases, you simply have a delimited token. '...' or $(...). After the trailing delimiter, scanning for the next token begins. Spaces are not needed to separated the tokens.

Except that in the case of hi"there" - as opposed to "hi"there - spaces are needed, given that it is interpreted as a single argument.

If a '.' appears after a completed argument (string literal, variable, etc) then it will be treated as a member accessor expression, with the constraint that there can be no space before the '.' and the preceding token.

While that makes perfect sense for an expression with a leading $, @, or (, (e.g., $Host.Name, $(Get-Date).Ticks, @(1, 2).Count, (Get-Date).Ticks, I question the value of doing it for string literals (e.g., 'foo'.bar).
Granted, you could do something like 'foo'.ToUpper(), but the likelihood that the user intent is to treat the . as a literal and conceive of the token as a multi-quoting-style string is much greater.

(Again, if 'foo' and .bar, due to being syntactically distinct elements, consistently became distinct arguments, that problem wouldn't arise.)

There is a fiddle that tries to stitch the two tokens "-parm:" and "value" back into a single string for literal arguments and there are requests to extend this fiddle. I'm beginning to think that would be a bad idea since the current hack seems to be causing people to make untrue assumptions. The more special cases there are the harder it is to explain.

I whole-heartedly agree with the sentiment about special cases.

Applying the hack consistently whenever something turns out not to be a named parameter reduces the number of special cases - at least from the user's perspective, which I think is more important.

The alternative is to disallow the use of such pseudo-parameters (except when calling external programs) and fail in their presence, informing users they need to use quoting.

By contrast, the current behavior of simply splitting the token into two arguments is useless.

Michael Klement (@mklement0)

Michael Klement (@mklement0): if this is purely a doc bug now, would you mind closing this issue so we can track everything in MicrosoftDocs/PowerShell-Docs#3038 ? Thanks!

Michael Klement (@mklement0)

Michael Klement (@mklement0)
I think I can explain all your "surprising aspects". (using Bruce Payette (@BrucePay)'s answer for some information) Note: the explanations are somewhat comprehensive so that this comment can be a stand alone description of the actual behaviour, not just a reply/analysis of your issue.

Asymmetry: Whether a quoted string or subexpression comes first matters:

Actually, the problem here is one of definition. The idea that command arguments are always separated by (un-escaped) white-space (as in bash) and the bits between the spaces are the individual "arguments" no longer applies when the command line is parsed according to a (complex) grammar where white-space is just one of several token separators/terminators. If a quoted string or subexpression is the first thing seen (in a space delimited "argument") then it is not the first part of the argument, it is an entire argument (after evaluation in expression mode). What follows is another argument (evaluated in expression or argument mode as appropriate). If the beginning of the "argument" (first token) does not invoke expression mode evaluation (e.g. the quoted string or subexpression is not "first") then the "argument" is parsed in argument mode. Even then, the "argument" might be terminated by something other than finding space (e.g. a grouping ( ) ). There is no "asymmetry" because in one case you have two syntactically separate arguments and in the other case you have one argument with parts that just look like the first case re-arranged (two apples vs one orange or maybe an apple and a kumquat vs an orange).

. following a leading quoted string is interpreted as [string] property access

This is actually a derivative of the previous one. The leading quoted string switches evaluation to expression mode. So when dealing with a string literal, which makes more sense, treating 'foo'.bar as a member access or as an automatic concatenation?
I would suggest that the answer depends upon which is the easiest to do in some other way (and then choose the meaning to be the other one).

If the user's intent was to create a concatenation of 2 literal strings, well that's trivial - 'foo.bar'
What about a non-literal? 'foo'.$x Again, that's easy - "foo.$x"
In each case it was even the same number of characters, just re-arranged slightly (and change ' into ").

Now, what if we really wanted to do a member access on a literal string, how else could we do that? It's not that difficult - $('foo').bar or $('foo').$x- but it is more complicated. So let's choose member access since it sometimes makes things just that little bit easier without increasing the difficulty the rest of the time.

Further, as Bruce Payette (@BrucePay) noted, the sequences $( ), " " and ' ' are all "delimited tokens". Initially I considered adding { } (script block) as it is a "delimited" token but its behaviour differs in the next section (argument not beginning with delimited token).

If an argument begins with a delimited token then it ends with the token unless immediately followed by a member access or indexer expression (.something with no spaces allowed or [ numeric expression ]) in which case it is (always) a member access / indexed token, which ends at the end of the member access / indexer expression unless immediately followed ... (and so on, recursively). Thus, the behaviour is consistent for all forms. Making (expandable or verbatim) string tokens act differently would create divergent behaviour (aka special case).

. as the initial char. followed by a quoted string results in 2 arguments

This one took a bit of research (and some guess work due, I suspect, to inadequate documentation). In fact, the "problem" is bigger than stated. A single . immediately followed by an expression of any kind results in separate arguments. Of course, this would be expected for grouping ( ) and script blocks { } but also applies to subexpressions $( ), strings (both types) and variable access (with or without member access). Why? Because (I think) a single . is being parsed as a call operator. Yes, I know it is referred to as the "dot sourcing operator" but this only applies when invoking powershell script files. . works exactly like & when the target is something other than a powershell script file. It just needs to be separated from its argument (to prevent it being treated as part of a file path). If its argument is a token of itself (such as a string or variable reference) then no space is necessary. Obviously, using . as a call operator is only meaningful if it is the first token in the command but apparently the token parser doesn't care where it is, just whether it satisfies the grammar for being treated as a separate token, i.e. NOT prepended to a bare word.

- as the initial char. in something that turns out not to be a parameter name results in 2 arguments with the presence of : or .
- as the initial char. causes embedded variable references not to be expanded

I suspect these two are different facets of the same "problem". By beginning an argument with -, I suspect that powershell prepares to form a named parameter argument and that's how it's parsed. If the "argument" matches a certain pattern (see below) then it is considered to be a parameter name (and possible value). The following regular expression defines what is considered to be a named parameter argument (for ASCII only, Unicode more complicated):

while ( $argstr = read-host ) {
  if ( $argstr -match '(-[_A-Za-z?][^"&''(,.:;[{|}]*(?::|(?=[ &(),.[{|}])))?(.*)' ) {
    "Named parameter: {0}`nRemainder: {1}" -f $matches[1..2]
  }
}

Interestingly, among the characters that terminate a parameter name, : is parsed into the parameter name while the rest start the next token. Even if powershell discovers that there is no parameter of that name, it still passes the separated name/value arguments (presumably so the invoked cmdlet/function can figure out what it wants to do), only restoring the original argument text if invoking a native command. Note: the same thing happens if a cmdlet/function is supplied with a splatting hash table with unexpected parameter names. The extra names and values appear in $args as -paramname: followed by the value in the next element.
Once powershell is expecting that an argument includes a parameter name specification, why would it apply variable expansion? This is another case of "which is easier to do another way". If you want programmatic control over which parameters get supplied, a splatting hash table is a better way (at least for invoking cmdlets/functions). If you allowed variable substitutions in parameter names then how do you specify a named parameter argument with a $ in its name (totally legal)

function contrived(${heybig$pender}) { echo ${heybig$pender} }

contrived -heybig$pender:$false

You could require the $ to be escaped but then it doesn't look like the same parameter name. I also point out (tongue firmly in cheek 😏) that programmatically adjusting parameter names is text manipulation not object manipulation (like splatting). Very 1990's and not powershelly at all but possibly necessary when invoking native commands because splatting is rather cmdlet/function centric (for native commands, just use expandable strings for the parameter names/values). An interesting, and perhaps even more "surprising", aspect of the "is it a parameter name" check is that, even if the pattern match fails, there is no backtracking. A variable substitution will be ignored up to the point where the pattern match fails but will occur if it appears after the point of failure

Write-Host -param$HOME'quoted'$HOME.length
-param$HOMEquotedC:\users\user.length

so it would seem that there are actually 3 modes when parsing a command line, expression mode, argument mode AND parameter name literal mode. The initial - selects parameter name literal mode and then the invalid parameter name character (' in the example) switches back to argument mode from that point.

And, of course, essentially none of the above is detailed in the documentation. Actually, what I found most lacking in the documentation (until I start researching this comment) was the explanation of the behaviour when the beginning of the argument is NOT an expression / parameter name literal mode invoking token, i.e. a bare word or direct variable access (no member access / indexer). In this case, it seems, the resulting argument is the concatenation of the value of every subsequent delimited token (not script blocks), direct variable access token (ignoring potential member access / indexer expressions) or bare word token (the value being the characters comprising it, between the end of the previous token and the beginning of the next token type) until a token other than one of these is found. Also, if the "value" of a subexpression is an array or maybe anything that implements [IList] (but not [IEnumerable] since hash tables aren't enumerated) then it is enumerated and all the elements converted to strings and joined by space. All this is to be gleaned from several unexplained examples. (Even this description is likely wrong or incomplete.)

Comments regarding opinions:

PowerShell will never behave exactly like bash, it's fundamentally different.
(In bash) The values of variables are expanded in place as strings. This could not work for PowerShell with live object(s).

Absolutely, in general, but in the case at hand - the compound tokens I describe - we are also dealing with strings - at least that's the only sensible interpretation.

The bit that was not stated by Bruce Payette (@BrucePay) in the above changes this a bit

The values of variables (which are all strings) are expanded in place as strings.

So yes, the compound tokens are strings BUT they are generated from the values of objects with many properties of varying types that need to be selected (individually or in groups) and converted to strings. Clearly a much more complicated grammar and evaluation process is required. Just assuming that the default interpretation of a (string like) construct is intended to be a (literal) string doesn't allow much versatility and could introduce complexity and confusion if that interpretation differed from the same construct's meaning elsewhere.

Granted, you could do something like 'foo'.ToUpper(), but the likelihood that the user intent is to treat the . as a literal and conceive of the token as a multi-quoting-style string is much greater.

I myself wouldn't think of it that way because just about everywhere else in powershell, that is syntactically similar, it would be a member access operator. Syntactically similar means a string with nothing stuck on the front of it. I suspect that just about everybody would look at

(nothing here touching the string) 'foo'.ToUpper()

and think, "Yep, member access", not "Um, member access unless maybe it is a command argument" (the second one being a fairly typical response to a "special case"). Just because a semantic grammar allows largely useless constructs doesn't mean those constructs should be repurposed to provide something more meaningful, particularly if there already exist equally effective ways to achieve the desired result. The useless constructs will simply not be used. Is this a waste? Think about this. A regular expression defines a set of strings that match the expression. How many of us actually generate every string from that expression (assuming it is a finite set)? We just use the expression to achieve a certain result. So it is here. The powershell grammar allows things like 'foo'.ToUpper() but that doesn't mean anybody has to use it. It's not a waste because the existence of the grammar (and thus 'foo'.ToUpper()) allows us to achieve our goal. The trick (as it relates to this thread) is to understand how the grammar works so that the difference between things like 'foo'.bar and bar.'foo' in a command argument is well known. To this end, the fewer "unless maybe"s, the better.

Also, you probably wouldn't do 'foo'.ToUpper() very often but you might like the ability to do

'Some really very long string that contains a whole lot of characters and might even extend over several lines'.ToCharArray()

Sure, you could do $([char[]]'Some really ... several lines'), which is similar, so how about

'55'.PadLeft(20,'#')

You'd have to enter (exactly) 18 #s. Tedious, much! What if the string were actually compound with unknown length

"$intpart.${fracpart}e$exppart".PadLeft(20,'~')

Granted, these are somewhat esoteric examples and the last one isn't a "constant" string but then how often would you want to auto concatenate 2 known strings. From the last example, would you want to split the behaviour up, not between strings ' '/" " and non-strings $( ) but between verbatim strings and the other (potentially non-constant) expandable strings and subexpressions?

(Again, if 'foo' and .bar, due to being syntactically distinct elements, consistently became distinct arguments, that problem wouldn't arise.)

So what about the syntactically distinct parts of

$Host.Name
$(Get-Date).Ticks
@(1, 2).Count
(Get-Date).Ticks

Is 'foo' just an object with properties like the example objects, or is it special?

(Just a note: your initial examples contain foo"bar" and bar"foo" which are syntactically identical. Did you mean "bar"foo?)

UberKluger

UberKluger:

The point of creating this issue was to point out that the existing behavior - even though it can be explained - is:

counterintuitive
governed by too many, inconsistent rules.

And, yes, in the initial post I meant to contrast foo"bar" and "bar"foo (now fixed), and their different behavior is a prime example of these issues.

Given that changing to a straightforward a compound-token-is-always-implicitly-an-expandable-string approach is not an option due to backward-compatibility concerns, the next best thing is to document these rules (when I initially created the issue, I was wondering whether at least some inconsistencies could be remedied as bucket 3 changes).

I've personally stopped contributing to this and the documentation repo, so I am unsubscribing from this thread, but if you feel you have an explanation of the current behavior that should be part of the documentation (or can at least serve as the basis for writing help content), I encourage you to create an issue in the documentation repo.

Navigation Menu

Repository navigation

Parsing of compound command-line tokens into arguments is surprising #6467

Asymmetry: Whether a quoted string or subexpression comes first matters:

`.` following a leading quoted string is interpreted as [string] property access

`.` as the initial char. followed by a quoted string results in 2 arguments

`-` as the initial char. in something that turns out not to be a parameter name results in 2 arguments with the presence of `:` or `.`

`-` as the initial char. causes embedded variable references not to be expanded

Environment data

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Parsing of compound command-line tokens into arguments is surprising #6467

Description

Asymmetry: Whether a quoted string or subexpression comes first matters:

. following a leading quoted string is interpreted as [string] property access

. as the initial char. followed by a quoted string results in 2 arguments

- as the initial char. in something that turns out not to be a parameter name results in 2 arguments with the presence of : or .

- as the initial char. causes embedded variable references not to be expanded

Environment data

Activity

BrucePay commented on Mar 22, 2018

mklement0 commented on Mar 26, 2018

joeyaiello commented on Oct 31, 2018

UberKluger commented on Jun 13, 2021

mklement0 commented on Jun 13, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`.` following a leading quoted string is interpreted as [string] property access

`.` as the initial char. followed by a quoted string results in 2 arguments

`-` as the initial char. in something that turns out not to be a parameter name results in 2 arguments with the presence of `:` or `.`

`-` as the initial char. causes embedded variable references not to be expanded