Select-String -SimpleMatch does only current culture search

We could enhance Select-String -SimpleMatch with Ordinal and OrdinalIgnoreCase search. It may speed up search in some scenarios up to 10x.

Also currently we can not switch to another culture if we use -SimpleMatch.

Note that without -SimpleMatch we use Regex and this new parameter(s) doesn't make sense.

Should we case-insensitive and ordinal by default? It makes sense for searching in logs?

Michael Klement (@mklement0)

Michael Klement (@mklement0) Could you please look the issue? How we could enhance parameter sets to expose more comparison types and perhaps cultures?
-Cuture en-US
-Comparison [Ordinal | OrdinalIgnoreCase | CurrentCulture | CurrentCultureIgnoreCase | SimpleCaseFolding ]

Michael Klement (@mklement0)

Michael Klement (@mklement0) Have you thoughts how we could design this?

Re cultures:

Support for arbitrary cultures directly via -Culture would follow the model of Compare-Object, Sort-Object, and Group-Object, which also use the current culture by default.

Like the latter, -Culture should accept culture names and also - obsolescent - LCIDs, which, unfortunately, is currently the only way to request the invariant culture (-Culture 127); -Culture '' should be accepted too, analogous to ``), but isn't.
That said, -Comparison InvariantCulture (see below) would be the less obscure way to request the invariant culture.

I have no sense of how much demand there is for being able to specify arbitrary cultures.

Re string-comparison modes:

Since Select-String has an existing -CaseSensitive switch, I would omit the case aspect and only support the following, with InvariantCulture added:

-Comparison [Ordinal | CurrentCulture | InvariantCulture | SimpleCaseFolding]

Note that CurrentCulture is the default.

-Comparison would have to be mutually exclusive with -Culture (the latter implying current-culture logic relative to the specified culture).

Michael Klement (@mklement0)

Michael Klement (@mklement0) Thank for great comment! Please look https://docs.microsoft.com/en-us/dotnet/api/system.stringcomparison?view=netcore-2.1

Ilya (@iSazonov)

My pleasure, Ilya (@iSazonov).

Good point re confusion with the case-sensitive-by-default-if-there's-no-IgnoreCase-suffix .NET StringComparison enumeration values.

To avoid that confusion, we'd have to allow all enumeration values (+ SimpleCaseFolding) and make -Comparison mutually exclusive with -CaseSensitive.

It's still somewhat awkward, however, because it contradicts PowerShell's case-INsensitive-by-default, must-opt-into-SENSITIVITY nature.

Considering the example https://docs.microsoft.com/en-us/dotnet/api/system.stringcomparison?view=netcore-2.1#examples
we could put -Comparison and -Culture in the same parameter set. Yes?

I don't think so, because:

Use of a specific culture with -Culture implies comparison mode CurrentCulture[IgnoreCase] (current meaning the specified culture in this case; in essence: you're asking for (fully) culture-sensitive comparisons).
Conversely, a combination such as -Culture ru-RU -Comparison Ordinal makes no sense.

Michael Klement (@mklement0)

That's certainly true Michael Klement (@mklement0), but there may be cases where you do want case-sensitive comparisons relative to a specific culture, which isn't really possible in your suggested setup.

Perhaps applying an argumentcompleter and a transform attribute to permit the use of Ordinal / Invariant / etc as though it were a true culture would be perhaps the most versatile approach?

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32):

which isn't really possible in your suggested setup.

It is, using the existing -CaseSensitive switch:

Select-String -Culture ru-RU -CaseSensitive ...

Your suggestion is interesting, but I wonder if it's too obscure, especially for users who aren't looking for a specific culture.

P.S.: Even with the suggested argument completer you'd then need a separate mechanism for opting into case-sensitivity (which -CaseSensitive could provide) - unless you're thinking that requesting a specific culture should imply case-sensitivity?

Not at all. The proposals here are a bit confusing, just trying to keep track. 😄

I see the -CaseSensitive vs -Comparison OrdinalIgnoreCase stuff has already been hashed out, I guess I missed it before.

a combination such as -Culture ru-RU -Comparison Ordinal makes no sense.

What about the API https://docs.microsoft.com/en-us/dotnet/api/system.string.compare?view=netcore-2.1#System_String_Compare_System_String_System_Int32_System_String_System_Int32_System_Int32_System_Globalization_CultureInfo_System_Globalization_CompareOptions_ ?

Ilya (@iSazonov)

Ilya (@iSazonov): From the linked page, emphasis added:

The comparison can be further specified by the options parameter, which consists of one or more members of the System.Globalization.CompareOptions enumeration. However, because the purpose of this method is to conduct a culture-sensitive string comparison, the CompareOptions.Ordinal and CompareOptions.OrdinalIgnoreCase values have no effect.

I don't want change to case-sensitive
Current default is CurrentCultureIgnoreCase. I'd prefer OrdinalIgnoreCase that is more fast, more expected and more useful for administrators I believe. It is my initial intention. And I hope the new Comparison parameter could resolve this.

I don't want change to case-sensitive

Neither do I. And I don't understand how your comment relates to my previous comment.

Are we in agreement, are we talking about different things, or did you mean to say something else?

As for what should have been the default (clearly, that ship has sailed): for consistency with most of PowerShell (regrettably, there are many exceptions), notably with -eq, it should have been InvariantCultureIgnoreCase.

Frankly, though, I'm unclear on when InvariantCultureIgnoreCase differs from OrdinalIgnoreCase. Do you know an example of where they differ (where linguistically appropriate behavior is exhibited by InvariantCultureIgnoreCase)?

The performance difference between the two is not large, fortunately: InvariantCultureIgnoreCase is only about 5 - 17% slower (on macOS 10.14, Ubuntu 18.04, Windows 10), from what I can tell (using Time-Command with 100,000 strings, averaged over 10 runs):

$a = (, 'hello') * 1e5; Time-Command { $a.Foreach({ $_.Equals('heLLo', 'InvariantCultureIgnoreCase')}) }, { $a.Foreach({ $_.Equals('heLLo', 'OrdinalIgnoreCase')}) }

And I don't understand how your comment relates to my previous comment.

Sorry, this was my response to:

I recommend against deviating from PowerShell's case-INsensitive default behavior

Frankly, though, I'm unclear on when InvariantCultureIgnoreCase differs from OrdinalIgnoreCase. Do you know an example of where they differ (where linguistically appropriate behavior is exhibited by InvariantCultureIgnoreCase)?

You can find a lot of useful in https://docs.microsoft.com/en-us/dotnet/standard/base-types/best-practices-strings
(Related #8064 - We definitely need to revise the code specially after #8566)
I believe the doc explain why I say about OrdinalIgnoreCase (not InvariantCultureIgnoreCase) for Select-String.

InvariantCultureIgnoreCase is only about 5 - 17% slower

My motivation to make Select-String faster is that my script parsed logs ~8 hours. I could save a few hours if the cmdlet was faster. And the cmdlet is significantly slower than native code utilities like grep. So getting 17% would be great!

So getting 17% would be great!

Absolutely; I think offering Ordinal[IgnoreCase] is helpful.
My point was that a (hypothetical) default of InvariantCultureIgnoreCase - in order to be consistent with most of PowerShell - wouldn't be so bad; though, based on the best-practices document you link to, it sounds like OrdinalgnoreCase is the better culture-agnostic choice anyway (as you've suggested):

Use StringComparison.Ordinal or StringComparison.OrdinalIgnoreCase for comparisons as your safe default for culture-agnostic string matching.

Choosing a consistent default across cmdlets and operators (#8064) is a great idea, but I wonder if that's considered an acceptable breaking change; while #8566 was indeed a breaking change, this was driven by unacceptable behavior having surfaced on Unix.

As an aside: I'm still confused about Ordinal vs. Invariant:

The linked docs state:

Do not use string operations based on StringComparison.InvariantCulture in most cases. One of the few exceptions is when you are persisting linguistically meaningful but culturally agnostic data.

I'd like to know (I haven't looked in earnest yet) what Invariant does that is linguistically meaningful.

For instance, the following baffles me:

'ı'.Equals('I', 'OrdinalIgnoreCase') # -> $true(!)
'ı'.Equals('I', 'InvariantCultureIgnoreCase') # -> $false

The roles seem to be reversed here: It is OrdinalIgnoreCase, not InvariantCultureIgnoreCase that recognizes the linguistically meaningful correspondence of the lowercase "dot-less" Turkish "i" ("ı") with its "dot-less" (ASCII-range) uppercase "I".

Michael Klement (@mklement0)

As an aside: I'm still confused about Ordinal vs. Invariant:

We can see OrdinalIgnoreCase implementation:
https://source.dot.net/#System.Private.CoreLib/shared/System/Globalization/CompareInfo.cs,559
where comparison is done in lowercase and difference in upper case.

But Invariant is always pinvoke to OS implementation. Seems every OS implements its own version of Invariant culture/locale. I guess it is a big problem because I don't found any standard for this. I think that for comparison and sorting Unicode standard defines Simple/Full Case Folding that is right thing for Invariant culture but it seems current implementation is different. Also Invariant culture contains definitions for date/time formats and it may vary between platforms too. I guess we could find Linux Invariant mapping tables in ICU library. For Windows I don't found any public docs only the related doc https://github.com/dotnet/corefx/blob/master/Documentation/architecture/globalization-invariant-mode.md

Michael Klement (@mklement0) if you still have questions we could ask for help MSFT expert Tarek Mahmoud Sayed.

Support for arbitrary cultures directly via -Culture would follow the model of Compare-Object, Sort-Object, and Group-Object, which also use the current culture by default.

Based on this and on #8824 make sense consider new -Comparison parameter for all these cmdlets?

That's probably a good idea.

Additionally, we'd need a different mechanism if we wanted to make the same functionality available to operators:

For string-comparison-based operators, it is conceivable to support CurrentCulture, Ordinal and SimpleCaseFolding too:

-eq, -lt, -gt, -contains, -in, switch (without -Wildcard or -Regex), ~~-split with the SimpleMatch option~~:
- From what I understand, these currently invariably use the invariant culture, correct?

By contrast, PowerShell's wildcards and .NET regexes appear to be limited with respect to linguistic comparisons due to seemingly operating on one-to-one character mapping only:

wildcard operators: -like, switch -Wildcard:
- PowerShell's wildcards use the current culture, but do not support linguistic comparisons due to doing character-by-character comparisons.
regex operators: (-match, -replace, -split, switch -regex)
- .NET regexes seemingly use the current culture for character-by-character case mapping, right?

For regexes, the only other mode that .NET offers is use of the invariant culture - not also ordinal comparison / simple case-folding.
Among the regex operators listed, it is only -split that already offers that, via its third RHS operand; e.g., 'a-b-c' -split '-', 0, 'CultureInvariant'.

By contrast, for wildcards, which are under PowerShell's control - and currently invariably use current-culture character-individual lowercase conversion - we can conceivably offer both the invariant culture and ordinal comparison / simple case-folding.

-eq, -lt, -gt, -contains, -in, switch (without -Wildcard or -Regex), -split with the SimpleMatch option:
From what I understand, these currently invariably use the invariant culture, correct?

I can not confirm. I did not found this in parserutils.cs.

You haven't found the source of the behavior, or you found evidence to the contrary?
I haven't looked at the code, but de facto it appears to be true (using ı (U+0131) as an example, which is the lowercase equivalent of I only in Turkish and Azeri):

[cultureinfo]::CurrentCulture = 'tr' # Turkish
([string] [char] 0x131).equals('I', 'CurrentCultureIgnoreCase') # $true
([string] [char] 0x131).equals('I', 'InvariantCultureIgnoreCase') # $false
([string] [char] 0x131) -eq 'I' # $false too

That it isn't ordinal comparison is evidenced by 'ss' -eq 'ß' returning $true (on Windows).

I considered only -split and not found that we set InvariantCulture.

In Binders.cs I see only InvariantCulture - all operators use seems invariant culture.

Navigation Menu

Repository navigation

Select-String -SimpleMatch does only current culture search #8180

4 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Select-String -SimpleMatch does only current culture search #8180

Description

Activity

iSazonov commented on Nov 16, 2018

iSazonov commented on Jan 18, 2019

mklement0 commented on Jan 30, 2019

iSazonov commented on Jan 30, 2019

mklement0 commented on Jan 30, 2019

iSazonov commented on Jan 31, 2019

mklement0 commented on Jan 31, 2019

vexx32 commented on Jan 31, 2019

mklement0 commented on Jan 31, 2019

mklement0 commented on Jan 31, 2019

vexx32 commented on Jan 31, 2019

iSazonov commented on Jan 31, 2019

mklement0 commented on Jan 31, 2019

4 remaining items

iSazonov commented on Feb 1, 2019

mklement0 commented on Feb 1, 2019

iSazonov commented on Feb 1, 2019

mklement0 commented on Feb 1, 2019

iSazonov commented on Feb 2, 2019

iSazonov commented on Feb 5, 2019

mklement0 commented on Feb 6, 2019

iSazonov commented on Feb 6, 2019

mklement0 commented on Feb 6, 2019

iSazonov commented on Feb 6, 2019

iSazonov commented on Feb 6, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions