Select-String where it can return multiple lines on a match, and this seem to depend on the size of the console window.

Steps to reproduce

Create a file named "logs.log"

2017-09-06T19:37:17.115+00:00 node2 2017-09-06 19:37:17,114 Level="WARNING" Name="support.pulse" Message="Irregular pulse duration detected" Duration="2.31735133001"
2017-09-06T21:20:27.829+00:00 node1 2017-09-06 21:20:27,827 Level="WARNING" Name="support.pulse" Message="Irregular pulse duration detected" Duration="6.652835013"
2017-09-06T21:21:04.163+00:00 node1 2017-09-06 21:21:02,207 Level="WARNING" Name="support.pulse" Message="Irregular pulse duration detected" Duration="2.33411307697"

Now I run (I would usually use a complex RegEx pattern to match, but this is added for simpicity):

Get-Content logs.log | Select-String -Pattern '6.6' | Out-File out.txt

Expected behavior

I would expect that in all cases, I get a file with one line

2017-09-06T21:20:27.829+00:00 node1 2017-09-06 21:20:27,827 Level="WARNING" Name="support.pulse" Message="Irregular pulse duration detected" Duration="6.652835013"

Actual behavior

Depending on the size of the console window when the command is run will depend on the number of lines in the file. If I run the console maximise (and on this system, I am running a 1920x1200 display, 100% font scaling), I get the desired result. However, if I window the console and run, I could get something like

2017-09-06T21:20:27.829+00:00 node1 2017-09-06 21:20:27,827 Level="WARNING" Name="support.pulse" 
Message="Irregular pulse duration detected" Duration="6.652835013"

That's two separate lines

Environment data

> $PSVersionTable
Name                           Value
----                           -----
PSVersion                      5.1.16353.1000
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.16353.1000
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

This is the documented behavior of Out-File. See the -Width parameter.

Get-Content logs.log | Select-String -Pattern '6.6' | Set-Content out.txt

performs the behavior you are expecting.

Mark Kraus (@markekraus)

Mark Kraus (@markekraus) thanks, however, simply running

Get-Content logs.log  | Out-File out.txt

Yields a normal width file - all the original lines are still on one line when you run the command in a windowed console. What is the difference after the string as been piped through Select-String?

Either way, this is confusing behaviour.

The difference is the type of object being processed. Select-String returns a Microsoft.PowerShell.Commands.MatchInfo (docs) where as Get-Content is returning Strings. The formatting is different.

This also produces the expected results:

Get-Content logs.log  | Select-String -Pattern '6.6' | %{$_.ToString()} | Out-File out.txt
# or 
Get-Content logs.log  | Select-String -Pattern '6.6' | %{$_.Line} | Out-File out.txt

Mark Kraus (@markekraus)

Thanks Mark Kraus (@markekraus) :)

It may be the documented behavior of Out-File, but I don't anyone who likes this behavior.

Jason Shirk (@lzybkr)

Jason Shirk (@lzybkr) Kind of. The documented behavior of Out-File is "somewhat" useful for "somewhat" emulating what the console would look like with the output. Most of the behavior complaints I have personally seen are addressed by using Set-Content instead. I always just assumed the purpose of Out-File was intended for preserving formatting.

In this particular instance, perhaps changing the default format behavior of Microsoft.PowerShell.Commands.MatchInfo makes sense (if possible). One would reasonably expect Select-String to return a String and for its output to behave like strings... even though it is documented that it returns MatchInfo or Bool.

This behavior seems to break the principle of least astonishment to me.

If we send anything as formatted output on screen how we can route the same in file? Is Out-File cmdlet designed for that? What is Out-* cmlets designed for?

IMO, it is Select-String that breaks the principle of least astonishment. Out-File has its purpose. Maybe the default behavior is debatable, but, that behavior is what distinguishes it from the *-Content cmdltes.

Select-String, on the other hand, should probably be returning Strings. If it did, the formatting would not result in Out-File line-wrapping or truncating. This is not to say returning MatchInfo is not useful, just not expected as the default behavior, IMO. The name implies it selects Strings. The output behavior could be changed but I have seen code that does rely on the MatchInfo objects. That behavior would need to be preserved (possibly with a switch parameter or something).

Select-String returning a MatchInfo is definitely useful for the usual reasons returning objects is useful. Here is one example - custom formatting to highlight the match: https://gist.github.com/lzybkr/dcb973dccd54900b67783c48083c28f7

Jason Shirk (@lzybkr)

Jason Shirk (@lzybkr) Absolutely! That's why I originally suggested possibly modifying the default format of MatchInfo instead of switching to String. If the default format of MatchInfo behaved like String, It would behave closer to what one would expect in most situations without breaking all the many things that have been made to work with MatchInfo.

Jason Shirk (@lzybkr)

The issue is not specific to Select-String (whose naming may be debatable, but there's only so much information you can cram into a cmdlet name, and it does operate on strings as input), it is fundamental to PowerShell's default output formatting:

Unfortunately - from what I can tell, without having dug into the source - all Out-* cmdlets use the same window width-sensitive formatting, even though respecting the host's window width really only makes sense for actual host (console) output (by default or via Out-Host).

Yes, the behavior may be documented, but as Jason Shirk (@lzybkr) and Steve Lee (@SteveL-MSFT) observe, it is undesirable / unexpected.
This is especially true for the > redirection operator, which is effectively an alias of Out-File.

Thus, the real fix - which is a breaking change, though hopefully a bucket 3 change - would be to remove any host window-width sensitivity from Out-* cmdlets that don't target the host (console), namely Out-File and Out-String.

A simpler demonstration of the problem:

# Default formatting truncates the property value and appends "..." to fit the window width.
> [pscustomobject] @{ str = 'x' * 1024 + '!' } 
str                                                                                                                                                                 
---                                                                                                                                                                 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...

# Unfortunately, the same truncation happens when you use Out-File, resulting in LOSS OF DATA:
> [pscustomobject] @{ str = 'x' * 1024 + '!' }  | out-file tmp.txt; Get-Content tmp.txt
str                                                                                                                                                                 
---                                                                                                                                                                 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...

As an aside: There seems to be a bug with respect to using > to redirect a [pscustomobject] instance to a file - see #4812.

If '>' is alias for Out-File can we replace it with Set-Content?

Unfortunately - from what I can tell, without having dug into the source - all Out-* cmdlets use the same window width-sensitive formatting, even though respecting the host's window width really only makes sense for actual host (console) output (by default or via Out-Host).

Right, which is what I have always assumed the Out-* commands are intended for. And yes, it does make sense, IMO, to have that functionality for files as well as the console. Sometimes you want the UX captured and Out-File accomplishes that goal

I don't see the point in making Out-File behave outwardly like Set-Content/Add-Content with a different internal implementation.

In defense of Out-File:
Out-File always did what I expected it to do when I was new to the language. What I saw on screen I wanted in a file and Out-File did just that. It wasn't until I was more familiar with the language and understood the "object" aspect of output (came from heavy text based languages) that I even saw this as a limitation.

Years ago I tried setting the default width to int.MaxValue - in the current implementation that kinda works in some places, but in others, it turns out to be a terrible idea - you will get padding to the width you asked for.

I do think we need to detect where the output is going - and not just for the width. If the output is to a file, we might also want to strip ansi escape sequences (with some sort of option to not do so).

As for performance - I lost any incentive to improve the perf of Select-String after finding ripgrep. It even has an option to output in a PowerShell friendly way so it wouldn't be too hard to write a wrapper and get objects, but I haven't done that yet.

CoreFX RegEx implementation is so bad?

I haven't compared the CLR regex to Rust's regex directly, but I'm confident we can do much better without touching the CLR's regex code.

To continue on the tangent, more generally:

The ability to call out to external utilities is important - even more so in the Unix world.
I'm sure there are other cmdlets whose performance with large input sets will necessitate using external utilities instead.

Conversely, other shells may want to take advantage of PowerShell's high-level features on demand, by calling PowerShell's CLI as needed.

The interfaces to properly support these scenarios aren't quite there yet / may never get there, unfortunately:

Problems with calling out of PowerShell (calling external utilities) - from what I understand, these will be fixed:
- Problems with POSIX-like globbing: overeagerness and inability to include hidden items.
- Lack of predictable argument passing (Unix) / "re-quoting" (Windows), with bash -c 'echo "hi there"' (prints just hi; double quotes in string literal arguments get removed when calling native commands #3049) and /bin/echo "a b\" (prints a b", Command line parsing bug: spaces and trailing backslash #4358) being two notable cases.
Problems with calling into PowerShell (powershell -command ....):
- PowerShell's nonstandard startup behavior is problematic, most notably that the profile is loaded by default.
- The above link lists all issues, but two are worth calling out, relating to the -Command and -File parameters:
  - They exhibit problematic interactive shell-like behaviors (with or without -NonInteractive) and lack consistent argument support - see CLI: pwsh -Command - and pwsh -File - unexpectedly exhibit pseudo-interactive behavior, lack support for argument passing #3223.
    - -File - - to provide a script via stdin (-) - is not officially supported and doesn't accept arguments (in addition to exhibiting problematic interactive-like behavior)
    - -Command - is officially supported, but by design doesn't accept arguments (and also exhibits problematic interactive-like behavior); see next point.
  - The way -Command currently parses the remaining arguments is problematic in itself and fundamentally incompatible with POSIX-like shells - see Command line arguments with a dollar sign #4024 (comment)
    - E.g., passing a simple string literal don't stored in a Bash variable to powershell -command as-is currently requires the following acrobatics:
      v="don't"; powershell -command Write-Output "'${v//\'/\'\'}'"
    - If the -Command argument were treated like an ad-hoc script to which arguments can be passed, as in POSIX-like shells (instead of reassembling all arguments into a single string that is then as a whole interpreted as PowerShell code, as currently happens), this would work as follows:
      v="don't"; powershell -command 'Write-Output $Args' "$v"
- The -Command issue will not be fixed so as to preserve backward compatibility, which I personally find very problematic; not sure about the others.

I went through ripgrep repo and see that it's a very young and single man's project. I saw a message that its public API should be re-designed. Also it still haven't library - I'd prefer the same way we use Newton.

I read ripgrep perf analize. It is veryinteresting. We can benefit from that. For example, make our FileSystem provider (dir traversal) more fast. It seems again related to migration on ETS.

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.

Navigation Menu

Repository navigation

Select-String where it can return multiple lines on a match, and this seem to depend on the size of the console window. #4767

Steps to reproduce

Expected behavior

Actual behavior

Environment data

17 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Select-String where it can return multiple lines on a match, and this seem to depend on the size of the console window. #4767

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment data

Activity

markekraus commented on Sep 7, 2017

swinster commented on Sep 7, 2017

markekraus commented on Sep 7, 2017

swinster commented on Sep 7, 2017

lzybkr commented on Sep 7, 2017

markekraus commented on Sep 7, 2017

SteveL-MSFT commented on Sep 10, 2017

iSazonov commented on Sep 10, 2017

markekraus commented on Sep 10, 2017

lzybkr commented on Sep 10, 2017

markekraus commented on Sep 10, 2017

mklement0 commented on Sep 12, 2017

iSazonov commented on Sep 12, 2017

markekraus commented on Sep 12, 2017

17 remaining items

lzybkr commented on Sep 13, 2017

iSazonov commented on Sep 14, 2017

lzybkr commented on Sep 14, 2017

mklement0 commented on Sep 14, 2017

iSazonov commented on Sep 15, 2017

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 23, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions