Add support for extension methods (LINQ)

Copied from TFS item 819298

Original Connect item: https://connect.microsoft.com/PowerShell/feedback/details/635454/add-support-for-linq

When LINQ is available, it can push queries into the native language of the underlying storage without requiring users to learn that query language. This transformation can speed up queries to perform hundreds times faster, and when the alternative is to ship large amounts of data over the wire to be filtered locally, thousands of times faster.

In many cases using LINQ can convert a script which ran so slow as to be infeasible into scripts that we can run every day, and in fact, can run whenever we like -- and it also drastically reduces CPU impact.

For a trivial example, see this post on Stack Overflow http://stackoverflow.com/questions/4559233/technique-for-selectively-formatting-data-in-a-powershell-pipeline-and-output-as where the switch to using LINQ (by embedded the LINQ into a C# function via Add-Type and calling it from PowerShell) provided 1000 times speedup. It's currently not possible to write that LINQ query at all in pure PowerShell, and the alternative (as it was written in the other answer on that StackOverflow question, using Where-Object) is extremely slow even on modest data sets.

The bottom line is that the Where-Object and Select-Object cmdlets are very slow, and using them to query databases, and other remote systems results in not only the slowdown of processing thousands of items in a Where-Object, but potentially sending them all over the wire as well.

Since LINQ to events is becoming available through the Rx project, it's more useful, and more important than ever, and we really need a way to write LINQ natively in PowerShell.

Oh please, oh please, oh please ... 😉

Staffan Gustafsson (@powercode)

(Staffan Gustafsson (@powercode) you'll probably be interested in this discussion).

Some random thoughts on what implications LINQ/extension methods have for PowerShell...

Joel Bennett (@Jaykul) I added an updated answer to your answer on StackOverflow that replaces all the pipeline use with .where(). This removes the last usage of the pipeline from the code. The result is pretty fast.

sergei (@vors) The example that Joel Bennett (@Jaykul) provided used XLinq which is perfectly usable from PowerShell as it stands. (XLinq isn't "really" LINQ anyway, it's just an alternative model for manipulating XML).

To see what the experience might be like, I've been playing with wrapping the [Linq.Enumerable] methods in ScriptMethods so I can write things like:

# Find processes with the most threads on this machine
(Get-Process).select{
    param($p) 
    [pscustomobject] @{
        Name=$p.Name; 
        Id=$p.Id; 
        Threads=$p.Threads.Count}}.
            OrderByDescending{param ($p) $p.threads}.
                Take(10)

The result is usually somewhat to much faster than the pipeline, even without lazy evaluation and a lot of copying. But it's also rather more complex to use. Now, while I am personally fond of this sort of thing, is this the experience we want to provide? For example, Michael Klement (@mklement0) seems adamantly opposed to pushing methods as a first-class experience. So perhaps we should have a real language experience like from $x where $_ .Size -gt 10 select .... Or a set of cmdlets that built up the query. Or even automatically rewrite parts of the pipeline into LINQ queries to make regular Where/Select/etc. automatically faster. Also note that there is a LINQ for PowerShell module on the gallery. It's incomplete and hasn't been updated in a couple of years (only has 1200 downloads) so maybe people really aren't that keen on LINQ in PowerShell after all.

And finally, what are the core scenarios for doing this? Is it just to be faster (a good thing)? Or is there specific functionality that you want to unlock. What about LINQ-to-SQL? We have classes now. Is this something we should invest in?

Another "stunt-query" that gets the 10 most common words from the text of "War and Peace". It uses a few extra operators I created (split(), trim() and match()) that aren't standard query operators.

$h=@{}
[io.file]::ReadAllLines("c:\temp\WarAndPeace.txt").
    trim().
        split().
            match('^[a-z]+$').
                aggregate{param ($x, $y) 
                    if ($x -isnot [hashtable]) { $h[$x]++; $x = $h } $x[$y]++; $x }.
                        select{param ($t) 
                            @($t.GetEnumerator()).
                                select{ param ($w)
                                    [pscustomobject] @{
                                        Word = $w.Key; 
                                        Count = $w.Value}}.
                                            OrderByDescending{$args[0].Count}.
                                                Take(10)}

Joel Bennett (@Jaykul)

So perhaps we should have a real language experience like from $x where $_ .Size -gt 10 select ....

To quote Joel Bennett (@Jaykul): Oh please, oh please, oh please ...

The query-like syntax fits better with the rest of PowerShell - a "whitespacey" experience that doesn't cause the confusion with argument mode that method syntax entails (though it's still an embedded language users will have to learn).

rewrite parts of the pipeline into LINQ queries to make regular Where/Select/etc. automatically faster.

Is that feasible? Certainly would be great.

And finally, what are the core scenarios for doing this?

As you say, a performance gain alone would be useful, but LINQ is such a wonderfully expressive, general-purpose feature that I can see its utility in a wide range of scenarios, including LINQ-to-SQL.

the confusion with argument mode that method syntax entails

I'm confused :-) What confusion? Method syntax is all expression mode. Every bit of the query I wrote above is in expression mode.

As you say, a performance gain alone would be useful,

Playing with this a bit more, I see that there will be a bound on how much faster we can get. LINQ query operators depend heavily on lambdas or, for us, scriptblocks, So the overhead of invoking a scriptblock will be a limiting factor on how fast we can get. (Perhaps things like PSLambda could help though at the risk of introducing another syntax mode. )

That's good to know; the PSLambda project sounds interesting; unless a solely method-based approach is chosen (see below), another syntax mode sounds unavoidable, though ideally it would feel like a variant of the existing (main) ones (argument mode and operators).

I'm confused :-) Method syntax is all expression mode.
Every bit of the query I wrote above is in expression mode.

And my head is still spinning.

To recap previous arguments:

Method syntax invites confusion with PowerShell's shell-like argument [parsing] mode (notwithstanding the fact that the option of using method syntax on .NET types directly is a great extensibility feature):

From the perspective of a shell user mostly familiar with command syntax (argument mode), the tricky differences are:

parentheses are required (vs. whitespace in argument mode)
arguments must be ,-separated (vs. whitespace in argument mode, where , constructs arrays)
bareword arguments (unquoted strings) aren't allowed

With the exception of the - little-known and little-used - .Where() and .ForEach collection "operators" (see PowerShell/PowerShell-RFC#126 for how to make them more PowerShell-like), PowerShell's own, core functionality relies only on cmdlets / functions (argument mode) and operators (expression mode, but resembling argument mode), and my vote is to keep it that way.

Michael Klement (@mklement0)

Michael Klement (@mklement0)

PowerShell's own, core functionality relies only on cmdlets / functions (argument mode)

I really don't understand where you got this idea from. It has never been the case that we relied only on cmdlets. There just wasn't enough time to cover all the scenarios (simple example: getting the substring of a string.) We targeted the most common scenarios with cmdlets because they gave the best user experience and then depended on .NET to back fill what remained.

and operators (expression mode, but resembling argument mode),

What do you mean by "resembling" here? They are distinct modes: barewords are allowed in argument mode but not in expression mode. Operators are allowed in expression mode but not in argument mode. They both allow simple expressions (including method calls) so I suppose that's a resemblance. Method calls are part of expression mode and I can't say I've ever seen someone try to use a method like a command but I've seem a lot of instances where command invocations are written like method calls (and subsequently don't work).

Bruce Payette (@BrucePay)

Bruce Payette (@BrucePay):

I really don't understand where you got this idea from.

To be clear: only you can provide behind-the-scenes glimpses - both historical and present - and I always appreciate them.

By contrast, I can only talk about de-facto behavior and guess at design intent in cases where it it isn't documented.

It has never been the case that we relied only on cmdlets.

De facto, virtually all of PowerShell's own functionality is provided by cmdlets and operators (with the exception of the useful, but little-known and little-used .ForEach() and .Where() collection "operator" methods that I wish had true operator representation.)

Whether it was the original intent or not, the following de-facto separation makes sense and is worth keeping:

Cover basic needs with PowerShell's own constructs - cmdlets and operators - whose syntax - while spanning two distinct parsing modes - doesn't clash:
- Cmdlets return are high-level, shell-syntax-friendly analogs to OO methods (loosely speaking).
- Objects returned by them are typically treated as "property bags" in basic command-line/scripting scenarios, meaning that only property access is needed.
- Objects and their properties can be acted on with operators in the context of expressions, without causing fundamental confusion (see below).
For advanced needs, provide full access to the .NET framework is available, which is a wonderful option to have and sets PowerShell apart from traditional shells; however, this option comes at a cost:
- You need to know more about the underlying .NET types and what methods they expose.
  - While Get-Member helps in discovering method signatures, with respect to getting help you're on your own (whereas the purpose of properties is usually self-evident).
- You need to know the subtleties of parameter binding when calling .NET methods directly.
- Last not least, you need to be aware of how method syntax differs from command syntax.

It therefore makes sense to limit introduction of new PowerShell-native features to cmdlets and operators.

simple example: getting the substring of a string.

Getting a substring via the .Substring() method is actually a good example of something that is rarely needed: due to PowerShell's OO nature and its excellent regex support via -split, -match, and -regex you typically do not need to deal with .Substring().

What do you mean by "resembling" here?

Understanding PowerShell's two fundamental parsing modes - argument mode vs. expression mode - is a challenge any PowerShell user faces, but it's the unavoidable price to pay for a shell that also offers the constructs of a "real" programming language (again, kudos to PowerShell).

So, yes, operators belong to the real of expressions, where string literals cannot be barewords (unquoted), for instance.

Aside from that, however, operators being separated from their operands by whitespace (rather than (...) and , - though , has a role in some operators too) is what I meant by "resembling" argument mode.
A negative way of phrasing it: Using literals (quoting need aside) and variable access - including property access on variables - doesn't clash with argument-mode syntax the way that method calls do.

$arr = 'Call', 'me',  'Ishmael'

# Operator syntax - reminiscent of argument mode (whitespace as separator, no (...))
$arr -join ' '

# Method syntax - uses (...) and "," - both of which function differently in argument mode.
[string]::Join(' ', $arr)

I've seem a lot of instances where command invocations are written like method calls (and subsequently don't work).

Indeed, and that's why introducing method calls for basic PowerShell functionality is problematic:

In the most benign case, you can get away with confusing the parsing modes, but the underlying misconception may come back to bite you later:

# The web is full of examples of confused syntax like this:
New-Object Foo.Bar($arg1, $arg2) # should be: New-Object Foo.Bar [-ArgumentList] $arg1, $arg2

The introduction of Set-StrictMode -Version 2 is living proof that the confusion is real:

function foo { param($bar, $baz) "[$bar], [$baz]" }
foo(1, 2) # binds *array* 1, 2 to $bar only - BREAKS with Set-StrictMode -Version 2 or higher.

On a related note, the PSv3 introduction of argument-mode-syntax comparison statements / operation statements to Where-Object / ForEach-Object is a nod to how argument-mode syntax is easier to type and read.

Michael Klement (@mklement0)

Michael Klement (@mklement0)

De facto, virtually all of PowerShell's own functionality is provided by cmdlets and operators

Except for little things like the entire language (statements, etc.). And types (casts). And the ability to call methods. If we hadn't wanted people to use methods we wouldn't have implemented the feature. And we did think a lot about the feature before adding it. The intended experience is not binary, it's a continuum. As you grow with experience, your abilities will also grow. To quote Jeffrey, PowerShell has an extremely wide dynamic range.

BTW: when I say types, I'm talking about things like this extremely old joke:

-join [char[]] ([int[]] [char[]] 'hal'| foreach {$_+1})

substring ... a good example of something that is rarely needed:

Hmmm ... I get 2.9 million hits for "powershell substring". Unfortunately. But you're right - we should have cmdlets for basic data manipulation . Please support Friedrich Weinmann (@FriedrichWeinmann)'s issue Commands for string methods & operators on the pipeline

Your example

$arr = 'Call', 'me',  'Ishmael'

is all expression mode; whitespace is unnecessary as in

$arr-join' '

And finally

($arr -join ' ').Count
[string]::Join(' ', $arr).Count

All in expression mode, parens are required around both to get the count.

Indeed, and that's why introducing method calls for basic PowerShell functionality is problematic:

No, that's not problem. The people who make these mistakes are invariably programmers who are used to methods in other languages and try to write PowerShell like it was C# (or whatever). They may not even be aware that PowerShell also has methods.

(You know, it just occurred to me how ironic it is to be having a discussion about how methods are bad in an issue proposing the addition of extension methods.:-))

Bruce Payette (@BrucePay)

Bruce Payette (@BrucePay):

Except for little things like the entire language (statements, etc.). And types (casts).

Yes, I didn't mention the language constructs explicitly, but I did say this:

Understanding PowerShell's two fundamental parsing modes - argument mode vs. expression mode - is a challenge any PowerShell user faces, but it's the unavoidable price to pay for a shell that also offers the constructs of a "real" programming language.

This should tell you that my concern is not about expression mode / control-flow statement syntax per se - on the contrary.

And:

For advanced needs, provide full access to the .NET framework is available, which is a wonderful option to have and sets PowerShell apart from traditional shells; however, this option comes at a cost:

This should tell you that your saying "If we hadn't wanted people to use methods" is refuting a point that I never made.

it's a continuum

With methods, it is not, for the reasons stated (syntax confusion, parameter-binding pitfalls, separate knowledge domain).
But - again - it's a is wonderful advanced option to have.

Please support Friedrich Weinmann (@FriedrichWeinmann)'s issue Commands for string methods & operators on the pipeline

Looks like a great idea (I'd already up-voted it).

BTW: when I say types, I'm talking about things like this extremely old joke:

I get the joke, but I don't know what you mean re types.

Your example $arr = 'Call', 'me', 'Ishmael' is all expression mode; whitespace is unnecessary

Just to drive the point home: my arguments were never about expression mode per se. They were about method syntax and the additional domain of knowledge you enter when using them (and, again, that is a wonderful advanced option).

Yes, you can omit whitespace (as you can in part in argument mode, Get-Item /|select name), but I hope we can agree that it is ill-advised, because the easy readability of of commands an operator-based expression comes from using whitespace).

The people who make these mistakes are invariably programmers who are used to methods in other languages and try to write PowerShell like it was C# (or whatever).

Indeed, you can use PowerShell entirely as if it were C# - virtually all the tools are there. You can get away without ever calling a cmdlet or use PowerShell's (unique to it) operators.
But I hope it is self-evident that that wouldn't be a good idea and that there's a PowerShell way to do things, and if that PowerShell way functions consistently with the fewer concepts to master the better, learning it is easier.

For someone who doesn't come from a programming background, argument mode (command-line syntax) and expression mode with properties is a manageable entry point, with an option to later branch out into the method world, as needed.

They may not even be aware that PowerShell also has methods.

With the exception of .Where() and .ForEach(), PowerShell itself has no user-facing methods that I am aware of (I'm not talking about the API), and my suggestions is that we keep it that way for a clean separation between PowerShell's own core functionality and the realm of .NET in general, which avoids conceptual and syntax challenges especially for beginners.

For people who want and need method calls, the door is wide open and - have I mentioned this before? - it is a great option to have.

(You know, it just occurred to me how ironic it is to be having a discussion about how methods are bad in an issue proposing the addition of extension methods.:-))

Let me know if you still feel that I've ever made the point that "methods are bad".

There's also no need to choose in this particular case; just like C#, we could provide _method syntax and query syntax.

In the case of method syntax, however, to the user this would be just another door into the realm of the underlying .NET world (no matter how much work is needed behind the scenes to "pass the functionality through").

By contrast, query syntax, such as your from $x where $_ .Size -gt 10 select ... example, with its whitespace-based token separation and absence of parentheses and lambdas, feels like a more natural PowerShell fit to me.

On a meta note: I feel that a lot of effort in this debate was spent on not discussing the actual points that I made. Were there specific ways in which I presented my arguments that obscured their intent?

I recently wrote a module (EFPosh) that exposes Linq functionality through cmdlets in a PowerShell-friendly way I think. I wanted to post it here and see what people thought.

Instead of trying to figure out how to write the Func delegates in PowerShell using .Net methods, I decided to handle that on the backend and expose query functionality through these commands:

New-EFPoshQuery
Start-EFPoshQuery
Add-EFPoshQuery

A query would go like this (pseudo-code as I wrote this for EntityFramework):

New-EFPoshQuery -Object $ObjectToQuery
Add-EFPoshQuery -Property '<TabComplete list of properties>' -Equals 'Value' -And
Add-EFPoshQuery -Property '<TabComplete list of properties>' -GreaterThan 5 -Or
Add-EFPoshQuery -Property '<TabComplete list of properties>' -Contains @('Array')
Start-EFPoshQuery -ToList -Distinct

The idea is once New-EFPoshQuery is executed, tab complete works for Add-EFPoshQuery through ArgumentCompletors. This way the syntax works from the command line and through a script.

It's a little weird to think about not executing the query in the line you start it, but deferred execution is an IQueryable idea that can come to PowerShell also!

On the back end I'm using the DynamicLinq library but that is more out of wanting to get something out the door rather than a necessity. It wouldn't be difficult (and might be easier) to build the delegates without a 3rd-party library.

Here's the code if you want to see what I'm doing:

https://github.com/Ryan2065/EFPosh/tree/master/src/Module/EFPosh/Commands

GitHub
Ryan2065/EFPosh
Contribute to Ryan2065/EFPosh development by creating an account on GitHub.

I was thinking about it, and in my module I store the IQueryable internally and let you build off of it and it's disposed when Start is run or New is re-run. Another way to write this that would be more PowerShell Friendly is this:

$Results = New-EFPoshQuery -Object $ObjectToQuery |
Add-EFPoshQuery -Property '<TabComplete list of properties>' -Equals 'Value' -And |
Add-EFPoshQuery -Property '<TabComplete list of properties>' -GreaterThan 5 -Or |
Add-EFPoshQuery -Property '<TabComplete list of properties>' -Contains @('Array') |
Start-EFPoshQuery -ToList -Distinct

So return the IQueryable and pipe it to the next command. That's more Poshey and allows you to write the query on multiple lines.

Joel Bennett (@Jaykul)

I generally agree with ideas behind this (dated) purpose but I also feel (from StackOverflow inquiries) that a lot of PowerShell scripters do not completely understand the actual differences between the PowerShell Pipeline and a deferred execution. They are not able to draw deeper conclusions from performance measurements (using Measure-Command). Meaning that they just want to use linq in their script because they're told it is up to a 1000 times faster or simply because they see a few methods were it appears that one takes 1000 times more CPU cycles than the other. This often leads to twisted Linq/PowerShell implementations sometimes even combined with a the addition assignment operator (+= ) to build an object collection.
Besides, so far I can tell, a Linq query on the PowerShell pipeline "InputObject" items will always require to hold the pipeline.
In other words, I think it is important to stay realistic, a "1000 times speedup" (implying the PowerShell pipeline is quiet worthless), is practically not what you get as a performance improvement by using Linq methods and measuring the performance of the whole end-to-end PowerShell solution.

Taking Joel Bennett (@Jaykul) measurements as an example.
Where it is assumed that $InputObjects already resides in memory, which is often not the case as it usually comes from somewhere else (e.g. a disk, AD, remote database, the cloud,...) and also requires a strict type (were usually PowerShell variables are loosely typed). It is important to look at the performance of retrieving the actual results of the concerned methods too:

Include result measurements

Write-Host "Initializing Collection"
[System.Collections.Generic.KeyValuePair[string, int][]]$InputObjects = (1..500000).ForEach({ [System.Collections.Generic.KeyValuePair[string, int]]::new("$_", $_) })

Write-Host "Invoking: Where-Object"
    $Measure = Measure-Command {
        $Pipeline = $InputObjects | Where-Object { $_.Key -eq "5000" }
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"

Write-Host "Invoking: .Where()"
    $Measure = Measure-Command {
        $Where = $InputObjects.Where({ $_.Key -eq "5000" })
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"

Write-Host "Invoking: [System.Linq.Enumerable]::Where()"
    $Measure = Measure-Command {
        $Linq = [System.Linq.Enumerable]::Where($InputObjects, [Func[[System.Collections.Generic.KeyValuePair[string, int]],bool]]{ param($x) $x.Key -eq "5000" })
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"

Write-Host "Invoking: [Linq.Enumerable]::Where with PSLambda"
    $Measure = Measure-Command {
        $PSlambda = New-PSDelegate { param($x) $x.Key -eq "5000" } -Delegate ([Func[[System.Collections.Generic.KeyValuePair[string, int]],bool]])
        $Lambda = [System.Linq.Enumerable]::Where($InputObjects, $PSlambda)
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"
    
Write-Host "Retrieving: Where-Object"
    $Measure = Measure-Command {
        $PipelineValue = $Pipeline.Value
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"

Write-Host "Retrieving: .Where()"
    $Measure = Measure-Command {
        $WhereValue = $Where.Value
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"

Write-Host "Retrieving: [System.Linq.Enumerable]::Where()"
    $Measure = Measure-Command {
        $LinqValue = $Linq.Value
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"

Write-Host "Retrieving: [Linq.Enumerable]::Where with PSLambda"
    $Measure = Measure-Command {
        $LambdaValue = $Lambda.Value
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"

In a fresh PowerShell session this yields:

Initializing Collection
Invoking: Where-Object
ExecutionTime: 3.9269345

Invoking: .Where()
ExecutionTime: 1.0701013

Invoking: [System.Linq.Enumerable]::Where()
ExecutionTime: 0.0144116

Invoking: [Linq.Enumerable]::Where with PSLambda
ExecutionTime: 0.0985589

Retrieving: Where-Object
ExecutionTime: 0.0033488

Retrieving: .Where()
ExecutionTime: 0.005539

Retrieving: [System.Linq.Enumerable]::Where()
ExecutionTime: 1.7921827

Retrieving: [Linq.Enumerable]::Where with PSLambda
ExecutionTime: 0.0645393

It is important to look at the performance of retrieving the actual results of the concerned methods too:

Note that the above examples do take this into account as well. While Measure-Command does output to a null pipe, it does still enumerate output. It just doesn't send it anywhere. Though of course it is still valuable to see the timing of the distinct steps, just want to make sure it's known since it's not particularly obvious.

Where it is assumed that $InputObjects already resides in memory, which is often not the case as it usually comes from somewhere else (e.g. a disk, AD, remote database, the cloud,...)

If we had true Linq, we could use Entity Framework and do server-side processing of our Linq queries with DB providers so the whole collection isn't brought back and evaluated in memory.

If we had true Linq, we could use Entity Framework and do server-side processing of our Linq queries with DB providers so the whole collection isn't brought back and evaluated in memory.

Support for IQueryable is a whole different host of work that probably needs it's own issue.

I mean, Linq works with both IQueryable and IEnumerable. Why can't a solution hit both targets?

EFPosh has a "BinaryExpressionConverter" to turn PowerShell binary expressions like $x -eq 1 into Linq binary expressions. I used that in the above tests and it was by far the quickest, also works with IQueryable and IEnumerable.

I left the Import statement to show where the dll came from

Import-Module "C:\Users\RyanEphgrave\OneDrive - Eph-It\Documents\WindowsPowerShell\Modules\EFPosh\1.2.6\Dependencies\BinaryExpressionConverter.dll" -Force

Write-Host "Initializing Collection"
[System.Collections.Generic.KeyValuePair[string, int][]]$InputObjects = (1..500000).ForEach({ [System.Collections.Generic.KeyValuePair[string, int]]::new("$_", $_) })

Write-Host "Invoking: Where-Object"
    $Measure = Measure-Command {
        $InputObjects | Where-Object { $_.Key -eq "5000" }
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"

Write-Host "Invoking: .Where()"
    $Measure = Measure-Command {
        $InputObjects.Where({ $_.Key -eq "5000" })
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"

Write-Host "Invoking: [System.Linq.Enumerable]::Where()"
    $Measure = Measure-Command {
        [System.Linq.Enumerable]::Where($InputObjects, [Func[[System.Collections.Generic.KeyValuePair[string, int]],bool]]{ param($x) $x.Key -eq "5000" })
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"

Write-Host "Invoking: [Linq.Enumerable]::Where with PSLambda"
    $Measure = Measure-Command {
        $lambda = New-PSDelegate { param($x) $x.Key -eq "5000" } -Delegate ([Func[[System.Collections.Generic.KeyValuePair[string, int]],bool]])
        [System.Linq.Enumerable]::Where($InputObjects, $lambda)
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"

Write-Host "Invoking: [Linq.Enumerable]::Where with BinaryExpressionConverter"
    $Measure = Measure-Command {
        $ConvertedExpression = ConvertTo-BinaryExpression -FuncType $InputObjects[0].GetType() -Expression { $_.Key -eq "5000" }
        [System.Linq.Enumerable]::Where($InputObjects, $ConvertedExpression.Compile())
    }
    Write-Host "ExecutionTime: $($Measure.TotalSeconds)`r`n"

Initializing Collection
Invoking: Where-Object
ExecutionTime: 8.3685408

Invoking: .Where()
ExecutionTime: 2.3706274

Invoking: [System.Linq.Enumerable]::Where()
ExecutionTime: 4.1634949

Invoking: [Linq.Enumerable]::Where with PSLambda
ExecutionTime: 0.1275231

Invoking: [Linq.Enumerable]::Where with BinaryExpressionConverter
ExecutionTime: 0.0232382

It's something to look at. I've had a lot of difficulties with this and I think it's mostly a lack of internal PowerShell knowledge, not that it can't be done. I spent weeks trying to figure out how to do simple things, like get variable values inside the Binary Expression.

I mean, Linq works with both IQueryable and IEnumerable. Why can't a solution hit both targets?

Because one is just adding binding support for extension methods and the other is translating PowerShell language into an expression tree usable by IQueryable (PowerShell already does this, and so does PSLambda, but neither are as exact as they'd need to be for this).

But I'm not saying it's not possible or shouldn't be done, just that it is it's own discussion and requires a lot more design work than just making some static methods appear like they're instance methods.

EFPosh has a "BinaryExpressionConverter" to turn PowerShell binary expressions like $x -eq 1 into Linq binary expressions. I used that in the above tests and it was by far the quickest, also works with IQueryable and IEnumerable.

Yeah that'll definitely top even PSLambda. It's binary expression calls out to LanguagePrimitives.Compare (which is a decision I would definitely not make twice)

It's something to look at. I've had a lot of difficulties with this and I think it's mostly a lack of internal PowerShell knowledge, not that it can't be done. I spent weeks trying to figure out how to do simple things, like get variable values inside the Binary Expression.

FWIW you can always see how Roslyn does things by looking at sharplab. That's something I learned way too late to take much advantage of.

Michael Klement (@mklement0)

Average user here just chiming in with my experience using LINQ tonight.

I had a 500k int[] and a 150k int[] and I needed to see which items in the 500k weren't part of the 150k. I let this run for several minutes while I looked for something better:

$c = $a.where{$_ -notin $b}

Then I ran this:

$c = [system.linq.enumerable]::Except([int[]]$a,[int[]]$b)

Which completed in under 2s.

I don't know why I needed to specify int[] for each item in there, and there is no documentation that I can find that specifically tells you do similar (I was referred here by Michael Klement (@mklement0)'s SO Answer).

If it was easier and/or better documented how to use powershell to do these LINQ queries, I think they'd be well loved by the community.

Szeraax

Szeraax ,

Which completed in under 2s.

Be careful with justifying PowerShell against LINQ. Linq is definitely fast but is almost the opposite of PowerShell (pipeline): where Linq uses a deferred execution / lazy evaluation (meaning that the statement [system.linq.enumerable]::Except([int[]]$a,[int[]]$b) isn't doing much and only executed when it is needed (e.g. display the results) and the PowerShell Pipeline let's you proactively work ahead (and input/process/output each item in a stream), e.g.: $a |Where-Object {$_ -notin $b } |....

I don't know why I needed to specify int[]

PowerShell is a loosely typed Programming Language by nature and Linq requires a strict type command

Quote PowerShell scripting performance considerations:

PowerShell scripts that leverage .NET directly and avoid the pipeline tend to be faster than idiomatic PowerShell. Idiomatic PowerShell typically uses cmdlets and PowerShell functions heavily, often leveraging the pipeline, and dropping down into .NET only when necessary.

**Note""
Many of the techniques described here are not idiomatic PowerShell and may reduce the readability of a PowerShell script. Script authors are advised to use idiomatic PowerShell unless performance dictates otherwise.

Taking your commands: for the linq query you can't just do c$[3] but need to do @(c$)[3] where you actually might also notice the deferred execution as (the first time) this takes a longer on the variable $c provided by the linq query then the one resulted from PowerShell.
See also: Comparing two arrays & get the values which are not common and Fastest Way to get a uniquely index item from the property of an array

Specific to your question, I would rather see propose #2132 implemented and have sweet PowerShell syntax like:

$c = $a -notin $b

Apparently we have to "bump" things we care about every 6 months now. Bump! Bump! Bump! 😉

I’m interested in the potential archeological significance of this item. Does anyone know when Connect item 635454 was originally created?

You know how going back to old code always makes you facepalm with stuff like, "wait, that's what I thought the best way was??"

I didn't expect the same facepalm from looking at a Github issue. I've learned a lot about LINQ and PowerShell since my comment wayyy back when. :) Maybe this provides some useful context on what this issue feels like for me to the internet archeologists that come along in the future to look.

Navigation Menu

Repository navigation

Add support for extension methods (LINQ) #2226

23 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add support for extension methods (LINQ) #2226

Description

Activity

Jaykul commented on Sep 19, 2016

BrucePay commented on Apr 30, 2018

BrucePay commented on Apr 30, 2018

mklement0 commented on Apr 30, 2018

BrucePay commented on Apr 30, 2018

mklement0 commented on Apr 30, 2018

BrucePay commented on May 1, 2018

mklement0 commented on May 2, 2018

BrucePay commented on May 2, 2018

mklement0 commented on May 3, 2018

23 remaining items

Ryan2065 commented on Jun 2, 2020

Ryan2065 commented on Jun 2, 2020

iRon7 commented on Jul 22, 2021

SeeminglyScience commented on Jul 22, 2021

Ryan2065 commented on Jul 22, 2021

SeeminglyScience commented on Jul 23, 2021

Ryan2065 commented on Jul 23, 2021

SeeminglyScience commented on Jul 23, 2021

Szeraax commented on Dec 6, 2021

iRon7 commented on Dec 6, 2021

Jaykul commented on Oct 30, 2023

TooManyLeaves commented on Mar 31, 2025

Szeraax commented on Mar 31, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions