OutOfMemoryException when running my PowerShell script

Question

Here's the PowerShell script I am using to add "segment99" to the beginning of all the text files (one by one) within a folder:

Set Environmental Variables:

$PathData = '<<ESB_Data_Share_HSH>>\RwdPnP'

Go to each text file in the specified folder and add header to the file:

Get-ChildItem $PathData -filter 'test_export.txt'|%{

$content = '"segment99" ' + [io.file]::ReadAllText($_.FullName)
[io.file]::WriteAllText(($_.FullName -replace '\.txt$','_99.txt'),$content)

}

This is giving me the following error:

Error: Exception calling "ReadAllText" with "1" argument(s): "Exception of type 'Syste
Error: m.OutOfMemoryException' was thrown."
Error: At D:\apps\MVPSI\JAMS\Agent\Temp\JAMSTemp13142.ps1:17 char:51
Error: + $content = '"segment99" ' + [io.file]::ReadAllText <<<< ($_.FullName)
Error:     + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
Error:     + FullyQualifiedErrorId : DotNetMethodException
Error:

I am running this code on a folder that has 20 files, each over 2 GB.

How can I fix this?

jon Z · Accepted Answer · 2011-11-15 08:35:28Z

3

Copying a header file + a large file to a new file will be less prone to outofmemory exceptions (for files of that size):

$header = '"segment99"'
$header | out-file header.txt -encoding ASCII
$pathdata = "."
Get-ChildItem $PathData -filter 'test_export.txt' | %{
  $newName = "{0}{1}{2}" -f $_.basename,"_99",$_.extension
  $newPath = join-path (split-path $_.fullname) $newname
  cmd /c copy /b "header.txt"+"$($_.fullname)" "$newpath"
}

edited Nov 15, 2011 at 8:35

answered Nov 15, 2011 at 6:34

jon Z

16.7k1 gold badge37 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

jon Z Over a year ago

@RomanKuzmin I forgot the encoding in the out-file cmdlet. Thanks for catching. Tested on a large file (> 2GB) and works reasonably well (< 1 min)

jon Z Over a year ago

Just to be clear, the encoding of the header file should match the encoding of the test_export.txt file.

Roman Kuzmin Over a year ago

I like this solution, it is presumably the fastest and this is important due to large files (about 2GB). But the header should be written like [IO.File]::WriteAllText('header.txt', $header), that is without new line after it (note: the original code inserts the header into the first line, not adds a new line).

Roman Kuzmin · Accepted Answer · 2011-11-15 07:23:37Z

2

This is not optimal code but it solves the task without reading all text to memory: it adds the header to the first line and then outputs other lines. Also, note that it does nothing if the input file is empty.

Get-ChildItem $PathData -Filter 'test_export.txt' | %{
    $header = $true
    Get-Content $_.FullName | .{process{
        if ($header) {
            '"segment99" ' + $_
            $header = $false
        }
        else {
            $_
        }
    }} | Set-Content ($_.FullName -replace '\.txt$', '_99.txt')
}

answered Nov 15, 2011 at 7:23

Roman Kuzmin

42.4k11 gold badges105 silver badges123 bronze badges

5 Comments

manojlds Over a year ago

This is what I was thinking of suggesting. Anyway, why the .{process{ line? A foreach-object would work I suppose?

Roman Kuzmin Over a year ago

Yes, ForEach-Object is the same (basically) as .{process{..}} but much slower. When it is about 2GB, it matters.

manojlds Over a year ago

Oh..what is this .{ notation here?

Roman Kuzmin Over a year ago

. is the operator "invoke in the current scope". . {} is "invoke the script block". Each script block may have begin, process, and end blocks. Our script block has the process. Thus, finally we get . { process{} }

manojlds Over a year ago

Thanks for the explanation...should have guessed it, seemed odd :)

Collectives™ on Stack Overflow

OutOfMemoryException when running my PowerShell script

2 Answers 2

3 Comments

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related