PowerShell Script Efficiency

Question

I use PowerShell as much as possible for quick and easy scripting tasks; A lot of times during my job I will use it for data parsing, log file sifting, or for creating CSV\Text files.

One thing I can't figure out is why it can be very inefficient to perform certain data\IO tasks. I figure it has to do with something under the hood with the way it handles Pipelines or just something I haven't understood yet.

If you take the following logic to generate ABC123 ids, compile it in PowerShell and execute it, it will take less than 1 minute to complete:

$source = @'
    public static System.Collections.Generic.List<String> GetIds()
    {
        System.Collections.Generic.List<String> retValue = new System.Collections.Generic.List<String>();
        for (int left = 97; left < 123; left++)
        {
            for (int middle = 97; middle < 123; middle++)
            {
                for (int right = 97; right < 123; right++)
                {
                    for (int i = 1; i < 1000; i++)
                    {
                        String tmp = String.Format("{0}{1}{2}000", (char)left, (char)middle, (char)right);
                        retValue.Add(String.Format("{0}{1}", tmp.Substring(0, tmp.Length - i.ToString().Length), i));
                    }
                }
            }
        }
        return retValue;
    }
'@
$util = Add-Type -Name "Utils" -MemberDefinition $source -PassThru -Language CSharp

$start = get-date
$ret = $util::GetIds()
Write-Host ("Time: {0} minutes" -f ((get-date)-$start).TotalMinutes)

Now take the same logic, run it through PowerShell without compiling as an assembly and it takes hours to complete

$start = Get-Date
$retValue = @()
for ($left = 97; $left -lt 123; $left++)
{ 
    for ($middle = 97; $middle -lt 123; $middle++)
    { 
        for ($right = 97; $right -lt 123; $right++)
        { 
            for ($i = 1; $i -lt 1000; $i++)
            { 
                $tmp = ("{0}{1}{2}000" -f [char]$left, [char]$middle, [char]$right)
                $retValue += ("{0}{1}" -f $tmp.Substring(0, $tmp.Length - $i.ToString().Length), $i)
            }
        }
    }
}
Write-Host ("Time: {0} minutes" -f ((get-date)-$start).TotalMinutes)

Why is that? Is there some sort of excessive type casting or inefficient operation I am using that slows down performance?

mjolinor · Accepted Answer · 2015-03-05 18:28:06Z

5

You're killing your performance right here:

$retValue += ("{0}{1}" -f $tmp.Substring(0, $tmp.Length - $i.ToString().Length), $i)

Array additions are a very "expensive" operation. What you're doing is basically creating a brand new array every time, composed of the original array plus the new element.

Edit: This kind of array addition is not only inefficient, but totally unnecessary. All you have to do is simply output those values to the pipeline, and assign the result back to the variable.

$start = Get-Date
$retValue =
for ($left = 97; $left -lt 123; $left++)
{ 
    for ($middle = 97; $middle -lt 123; $middle++)
    { 
        for ($right = 97; $right -lt 123; $right++)
        { 
            for ($i = 1; $i -lt 1000; $i++)
            { 
                $tmp = ("{0}{1}{2}000" -f [char]$left, [char]$middle, [char]$right)
                "{0}{1}" -f $tmp.Substring(0, $tmp.Length - $i.ToString().Length), $i
            }
        }
    }
}
Write-Host ("Time: {0} minutes" -f ((get-date)-$start).TotalMinutes)
Time: 1.866812045 minutes

edited Mar 5, 2015 at 18:28

answered Mar 5, 2015 at 18:15

mjolinor

68.7k7 gold badges118 silver badges141 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

The Unique Paul Smith Over a year ago

very nice! Makes a lot of sense now. I'm assuming System.Collections.Generic.List<String> is not the same type as @(). Does the System.Collections.Generic.List<String>.Add method perform array expansion? Thanks again!

mjolinor Over a year ago

The .add() method of generic lists is much more efficient than doing array addition. The arralist type seems to be the fastest in terms of efficiently adding elements, but generally you only need to do this in situations where the pipeline is already being used for other objects, and you need to do an incidental accumulation outside of the pipeline.

Collectives™ on Stack Overflow

PowerShell Script Efficiency

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related