Select-String not working on piped object using Out-String

Question

I am doing an API request which returns a bunch of data. In attempted to search through it with Select-String, it just spits out the entire value stored in the variable. This is an internet server which I am calling an api.

$return = Invoke-RestMethod -Method GET -Uri $uri -Headers @{"authorization" = $token} -ContentType "application/json" 
$file = $return.data
$file | Out-String -Stream | Select-String -Pattern "word"

this returns the entire value of $file. printing $file looks like same as the pipe output. Why is this not working?

$file.Gettype says it is a system.object, another answer said to use Out-String, but something is not working.

$file.Gettype
IsPublic IsSerial Name                                     BaseType                   
-------- -------- ----                                     --------                   
True     True     String                                   System.Object

There are quiet some holes in your arguments that it is not work the way you expected. Please create a minimal reproducible example and add an example of the $return.data to the question so that we can prove it does work (or not). — iRon
– iRon, Commented Jan 15, 2021 at 8:56

iRon · Accepted Answer · 2021-01-15 17:23:41Z

2

On closer inspection, I suspect that your issue comes from an ambiguity in the Out-String documentation:

`-Stream`

Indicates that the cmdlet sends a separate string for each line of an input object. By default, the strings for each object are accumulated and sent as a single string.

Where the word line should be read as item.

To split you raw string into separate lines, you will need to split your string using the following command:

$Lines = $return.data -split [Environment]::NewLine

Note that this assumes that your data uses the same characters for a new line as the system you working on. If this is not the case, you might want to split the lines using an regular expression, e.g.:

$Lines = $return.data -split "`r*`n"

So what does the-Stream parameter do?

It sends a separate string for each item of an input object.
Where in this definition, it is also a known common best PowerShell practice to use a singular name for possible plural input objectS.

Meaning if you use the above defined $Lines variable (or something like $Lines = Get-Content .\File.json), the input object "$Lines" is a collection of strings:

$Lines.GetType().Name
String[]

if you stream this to Out-String it will (by default) join all the items and return a single string:

($Lines | Out-String).GetType().Name
String

In comparison, if you use the -Stream parameter, it will pass each separated item from the $Lines collection directly to the next cmdlet:

($Lines | Out-String -Stream).GetType().Name
Object[]

I have created a document issue for this: #7133 "line" should be "item"

Note:

In general, it is a bad practice to peek and poke directly into a serialized string (including Json) using string methods and/or cmdlets (like Select-String). Instead you should use the related parser (e.g. ConvertFrom-Json) for searching and replacing which will result in an easier syntax and usually takes care of known issues and pitfalls.

edited Jan 15, 2021 at 17:23

answered Jan 15, 2021 at 12:51

iRon

24.4k10 gold badges60 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

arealhobo Over a year ago

You explanation is very informative and solved my issue. When the API returns the data, each new line has a line feed character. So splitting off by just using "`n" solved the issue with Select-String only returning one match or no matches at all.

mklement0 Over a year ago

Good answer, but the per-item explanation isn't correct - it's about in-band vs. out-of-band formatting - please see my answer; also, I suggest "`r?`n" or '\r?\n' as the regex.

mklement0 · Accepted Answer · 2021-01-20 02:57:28Z

To complement iRon7's helpful answer with the precise logic of Out-String's -Stream switch, as of PowerShell 7.1:

Out-String, like the other Out-* cmdlets such as Out-File, uses PowerShell's rich output-formatting system to generate human-friendly representations of its input objects.

Without -Stream, Out-String only ever produces a single, (typically) multiline string.
With -Stream, line-by-line output behavior typically occurs - except for input objects that happen to be multiline strings, which are output as-is.
- Because this exception is both obscure and unhelpful, GitHub proposal #14638 suggests removing it.

For so-called in-band data types, -Stream works as follows, which truly results in line-by-line output:

Input objects are formatted by PowerShell's rich formatting system, and the lines that make up the resulting representation are then output one by one.

Out-of-band data types are individually formatted outside of the formatting system, by simply calling their .NET .ToString() method.

In short: data types that represent a single value are out-of-band, and in addition to [string] out-of-band data types also comprise [char] and the various (standard) numeric types, such as [int], [long], [double], ...

[string] is the only out-of-band type that itself can result in a multiline representation, because calling .ToString() on a string is effective no-op that returns the string itself - whether it is single- or multiline.

Therefore:

Any string - notably also a multiline string - is output as-is, as a whole, and splitting it into individual lines requires an explicit operation; e.g. (note that regex \r?\n matches both Windows-style CRLF and Unix-style LF-only newlines):
```
"line 1`nline 2`nline 3" -split '\r?\n' # -> 'line 1', 'line 2', 'line 3'
```
If your input objects are a mix of in-band objects and (invariably out-of-band) multiline strings, you can combine Out-String -Stream with -split; e.g.:
```
((Get-Date), "line 1`nline 2`nline 3" | Out-String -Stream) -split '\r?\n' 
```

Dusty Vargas · Accepted Answer · 2021-01-15 05:24:25Z

0

Select-String outputs Microsoft.PowerShell.Commands.MatchInfo objects. It seems to me that the output is somehow fancified via the PS engine or something to highlight your match, but ultimately it does print the entire matched string.

You should check out the members of the object Select-String provides, like this:

$file | Out-String -Stream | Select-String -Pattern "word" | Get-Member

   TypeName: Microsoft.PowerShell.Commands.MatchInfo

Name               MemberType Definition
----               ---------- ----------
...
Matches            Property   System.Text.RegularExpressions.Match[] Matches {get;set;}
...

What you're interested in is the Matches property. It contains a bunch of information about the match. To extract exactly what you want, look at the Value property of Matches:

($file | Out-String -Stream | Select-String -Pattern "word").Matches.Value

word

Another way:

$file | Out-String -Stream | Select-String -Pattern "word" | ForEach-Object {$_.Matches} | Select-Object -Property Value

Value
-----
word

Or

$file | Out-String -Stream | Select-String -Pattern "word" | ForEach-Object {$_.Matches} | Select-Object -ExpandProperty Value

word

answered Jan 15, 2021 at 5:24

Dusty Vargas

9458 silver badges17 bronze badges

1 Comment

arealhobo Over a year ago

This seems to only get one match found, not all matches. Using ).matches.value or with the ForEach-Object. If I save the data to a file, and read it from a file it works perfectly fine. Just not with the variable pipe.

Collectives™ on Stack Overflow

Select-String not working on piped object using Out-String

3 Answers 3

`-Stream`

2 Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related