-OutVariable doesn't unwrap single-object output and creates [System.Collections.ArrayList] values rather than [System.Object[]]

^{Note: Also affects other common parameters: see #3773}

Steps to reproduce

$null = Get-Date -OutVariable ov; $ov.GetType().FullName

Expected behavior

Type name System.DateTime

Actual behavior

Type name System.Collections.ArrayList

That is, the single-object output stored in the variable targeted with -OutVariable received an array list rather than a scalar (the single-element collection wasn't unwrapped).

Also, it is type [System.Collections.ArrayList] that is always used, in contrast with the [System.Object[]] instances you get with regular assignments.

See also: http://stackoverflow.com/a/40666568/45375

Environment data

PowerShell v6.0.0-alpha (v6.0.0-alpha.15) on Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64
PowerShell v5.1.14393.693 on Microsoft Windows 10 Pro (64-bit; v10.0.14393)

IIRC it's been this way since version 1. Changing it now would be a breaking change.

Note that -OutVariable produce live collection, you can reference to it while pipeline is still working:

1..10 | echo -OutVariable a | % { "$a" }

Using not resizable [System.Object[]] instead of [System.Collections.ArrayList] would be bad performance choice.

The behavior is surprising, and possibly falls into the unlikely grey area bucket.

Tagging for committee review.

@PowerShell/powershell-committee reviewed this and agree with the expected result even though it is a breaking change

While I agree that it's weird that this doesn't behave like pipeline output....

The only thing that PowerShell does this collection/single behavior for is pipeline output

These have been collections since the very beginning.

You'll be potentially breaking every single use of these in the history of PowerShell.

Can we follow the process and RFC this?

I'm pretty sure you're going to light up the comments on an RFC for this kind of breaking change.

As an afterthought, given that I just stumbled upon it:

Tee-Object -Variable exhibits pipeline, logic, as expected - unlike -OutVariable:

Single-object output produces a scalar (unwraps the output collection):

# Capture output from a command that outputs a *single* object
PS> $null = Get-Item / | Tee-Object -Variable out; $out.GetType().FullName
System.IO.DirectoryInfo  # *scalar* was captured

Multi-object output creates an [object[]] array.

# Capture output from a command that outputs a *multiple* objects
PS> $null = Get-ChildItem / | Tee-Object -Variable out; $out.GetType().FullName
System.Object[]

Yeah, I called out that behavior in the RFC comments 😉

In fact, Tee-Object already has the behavior you're looking for: Unlike -OutVariable, Tee-Object doesn't populate it's variable until the end of the pipeline. Because of that, it's able to not output a collection when there's only one item, and it outputs an array when there is (not an ArrayList).

Joel Bennett (@Jaykul)

Good point, Joel Bennett (@Jaykul), thanks - it's been so long...

That I "rediscovered" this inconsistency after all the discussion is telling, however - it baffled me anew.

After all this time, is this still a desired feature?

Seems like RFC never reached an agreement, and thus was closed. But this issue was left opened, so I wonder...

As far as I could tell at the time with the RFC, any change to the existing behaviour would be controversial and the current behaviour is considered a feature by some. More importantly, it's public behaviour that users depend on, so I suspect it's something we can't change and we should close this issue.

It is a candidate for #6745 (in the sense that a hypothetical future PowerShell version not beholden to backward compatibility should reconsider this behavior).

PetSerAl

It seems that no-one picked up on what might be the basic reason for using an [ArrayList] (even for single objects),

To add the output to the variable, instead of replacing any output that might already be stored there, type a plus sign (+) before the variable name.

Using an (immutable) [Object[]] could result in a performance degradation as it would have to be copied to a new [Object[]] every time it was updated. PetSerAl did give some insight but the example usage would be rare compared to -OutVariable +name (IMO).

As for storing a single object as a scalar, there would be no guarantee that a single object output stored in -OutVariable would not subsequently have later output appended. As with the issue of single items from pipelines being stored as scalars but possibly being used later as arrays (hence the advice about using the array subexpression operator @() in such cases), it would be cumbersome to have to check whether a -OutVariable contained a scalar or collection, or at least it would have been before PS3.0 added indexing of scalars, thereby somewhat obscuring the difference. Still, "Once an [ArrayList], always an [ArrayList]".

Note: Tee-Object -Variable does not have the option to append with + and so can make assumptions about the intended use of the variable and produce a result identical to that which would be stored by a variable capturing the pipeline output from Tee-Object. Further, if a variable containing a scalar or an [Object[]] is used as an appending -OutVariable then it is first converted to an [ArrayList] of the same length (1 or many), whether or not anything is appended, thereby maintaining the "flat" collection in a consistent container after use of -OutVariable.

PS > 1,2 | Tee-Object -Variable tee1
1
2
PS > $tee1.GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Object[]                                 System.Array

PS > $tee1.Length
2
PS > @() | Tee-Object -Variable tee2 -OutVariable +tee1
PS > $tee1.GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     ArrayList                                System.Object

PS > $tee1.Count
2
PS >

(Tested on 5.1.18362.145)

UberKluger

Agreed with UberKluger, it's very sensible to have Pipeline-native accumulation supported in some way in Powershell; and currently -OutVariable +var is the only way to accumulate multiple separate outputs in a performant way, without dropping into .NET ArrayList syntax.

I don't think altering the behaviour on a singleton would be a problem, although there are potential pitfalls - if an iteration produces lists, and the first list is unwrapped into the same variable, then subsequent lists would be appended to the end of the first list instead of alongside it.

However, with respect to the other observation, Array vs ArrayList type, OutVariable behaviour should emphatically not be changed from appending to ArrayLists to appending to Arrays - that would be a huge performance hit for code correctly using the parameter. Arrays and ArrayLists are comparable for appending about 1000 elements, but once you're dealing with 40,000 items there's a factor of 12 difference (10 seconds vs 2 minutes), and 80,000 it's a factor of 24, etc. So the current behaviour is the difference between a script quite handily processing millions of items in a minute versus taking hours. (Of course, appending to an array is not a typical Powershell idiom - I'll get to that.)

I think at the very least, OutVariable behaviour should be very clearly documented - it's evident that that's not the case currently, I can only imagine there was some intention to transparently handle Arrays and ArrayLists the same way in earlier design stages that never came to fruition. Note of course that deliberate unwrapping of an ArrayList is very easy, $($var) will give us a Singleton for a one-element ArrayList and an Object[] otherwise.

With respect to the need for separate accumulations, as facilitated by OutVariable: Ideally, tasks can be broken down into separate pipelines so that separate accumulations can be performed each with a single pipeline, but some tasks don't lend themselves to this, particularly when there's variegated interdependent checks where a value from one output will influence another; in those circumstances, the alternatives without OutVariable are using Out-File, which loses type information and is disk bound (but at least can append in constant time, and clearly specifies a 'separate' output), or writing to the common output stream and performing a post-hoc Where filter, which requires a great deal more coordination and setting of metadata to facilitate a filter than would simply accumulating to separate variables.

If I had to design an alternative to supersede -OutVariable it'd be named streams - $var = Create-Stream; @(1,2,3) | Write-Stream $var etc. This maps to natural Powershell idioms (including writing values directly to the pipe) and reflects how people will often use Out-File naturally without the performance hit of writing to disk. Unlike an Out- cmdlet the OutVariable parameter allows output to pass through, which means in practice you have to deliberately redirect to $null if you just wanted it accumulated in the ArrayList; not a pretty idiom.

UberKluger

Agreed with UberKluger, it's very sensible to have Pipeline-native accumulation supported in some way in Powershell; and currently -OutVariable +var is the only way to accumulate multiple separate outputs in a performant way, without dropping into .NET ArrayList syntax.

You can collate pipeline objects pretty easily with neither of those:

$results = @(
    Get-ChildItem $folder1
    Get-ChildItem $folder2
    Get-ChildItem $folder3
)

This produces a flat array with all the results of the three commands in it.

This produces a flat array with all the results of the three commands in it.

As long as you can put all of the outputting lines together in one place...

IMO it's less common to not be able to do that, but it does happen. In those instances, I don't think the -OutVariable or equivalent usage of List types is a huge burden.

You can collate pipeline objects pretty easily with neither of those:

Yeah sorry, I could have been clearer. Absolutely, that way is the best approach in general where it fits the use case; it's what Powershell encourages you to use and is the idiom people should use by default (or rather, your flat array strategy is a particular implementation of the "just write to Output" Powershell default). When people don't, it's normally because they're expecting Powershell to work like some other language with which they're more familiar.

However, if you have three categories of thing, say line 1 went to $a, line 2 went to $b, line 3 went to $c, Out-Variable is the most efficient way of doing this. It's better to use three different pipelines, but sometimes the three outputs are interrelated enough that it's difficult to extricate easily, and outputting each from within the same loop is the most sensible way to keep the code simple and efficient.

Other strategies - like post-processing Where filters, or PSCustomObjects with separate properties for each output - make sense in specific cases (the former where it's easy to distinguish the outputs post-hoc, the latter where there's always 1-for-1 relationship between each output), but OutVariable is the only general accumulator offered by Powershell that's actually algorithmically efficient.

Navigation Menu

Repository navigation

-OutVariable doesn't unwrap single-object output and creates [System.Collections.ArrayList] values rather than [System.Object[]] #3154

Steps to reproduce

Expected behavior

Actual behavior

Environment data

The only thing that PowerShell does this collection/single behavior for is pipeline output

Can we follow the process and RFC this?

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

-OutVariable doesn't unwrap single-object output and creates [System.Collections.ArrayList] values rather than [System.Object[]] #3154

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment data

Activity

BrucePay commented on Feb 17, 2017

PetSerAl commented on Feb 26, 2017

lzybkr commented on Feb 27, 2017

SteveL-MSFT commented on Mar 2, 2017

Jaykul commented on Mar 16, 2018

The only thing that PowerShell does this collection/single behavior for is pipeline output

Can we follow the process and RFC this?

8 remaining items

mklement0 commented on Sep 12, 2018

Jaykul commented on Sep 12, 2018

mklement0 commented on Sep 12, 2018

ratijas commented on Mar 2, 2021

rjmholt commented on Mar 2, 2021

mklement0 commented on Mar 2, 2021

UberKluger commented on Jan 20, 2022

KingAl commented on Jan 29, 2022

vexx32 commented on Jan 29, 2022

Jaykul commented on Jan 30, 2022

vexx32 commented on Jan 30, 2022

KingAl commented on Jan 31, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions