Use set content to create multiple output files

Question

I have a PowerShell script:

(Select-String -path c:\*.txt -Pattern 'searchstring').Line | set-content d:\output.txt

It works fine but the output file is too big. What I would like is for it to output multiple files so output.txt is the same number of files as is being read in with the wild card.

So:

c:\1.txt
c:\2.txt
c:\3.txt

Would output the matched lines to:

d:\1.txt
d:\2.txt
d:\3.txt

have you looked at the .Filename property of the objects returned by Select-String? you can group on that and then output to a new file based on the items in each group. — Lee_Dailey
– Lee_Dailey, Commented Mar 8, 2022 at 20:31
@Lee_Dailey, to avoid ambiguity in general you'd have to group by .Path. I had considered a Group-Object solution, but it is more memory-intensive (and possibly also slower overall) than calling Select-String on each input file, and the solution isn't any shorter. — mklement0
– mklement0, Commented Mar 8, 2022 at 20:47
@mklement0 - the OP seems to indicate the source files will all be in one directory ... and the dest files will be in another. that seems to sidestep the need for -Path. [grin] however, i suspect you are correct about the likely RAM needs. i have noticed that piping from G-CI to S-S is notably slower than using S-S to grab the files directly, so it may not be slower to use G-O in that situation. — Lee_Dailey
– Lee_Dailey, Commented Mar 8, 2022 at 21:49
@Lee_Dailey, that's why I said in general. If you're interested in grouping matches by the specific files they came from, there's no reason to use FileName for grouping - it may work in this specific case, but you're asking for trouble in other cases. The speed tradeoff doesn't come from what you describe - there's virtually no difference between piping file-info objects vs. using -Path to a single Select-String call - but comes from requiring multiple Select-String calls. That is balanced against the processing (not just memory) overhead that grouping brings. — mklement0
– mklement0, Commented Mar 8, 2022 at 21:55
@mklement0 - thank you for the clarification ... i do appreciate it! [grin] — Lee_Dailey
– Lee_Dailey, Commented Mar 9, 2022 at 21:05

mklement0 · Accepted Answer · 2022-03-08 20:03:29Z

2

As Santiago Squarzon suggests, use Get-ChildItem to retrieve the target files, then call Select-String on them one by one:

Get-ChildItem c:\*.txt |
  ForEach-Object {
    if ($lines = ($_ | Select-String -Pattern 'SearchPattern').Line) {
      Set-Content -LiteralPath (Join-Path d:\ $_.Name) -Value $lines
    }
  }

edited Mar 8, 2022 at 20:03

answered Mar 8, 2022 at 19:56

mklement0

452k68 gold badges728 silver badges987 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

sb5917 Over a year ago

This worked perfectly (or appears to be). The script has been running for 24hrs now. Thank you very much.

mklement0 Over a year ago

I'm glad to hear it, @sb5917; my pleasure.

Collectives™ on Stack Overflow

Use set content to create multiple output files

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related