1

Performing URLdecoding in CGI hybrid script by powershell oneliner:

echo %C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA| powershell.exe "Add-Type -AssemblyName System.Web;[System.Web.HttpUtility]::UrlDecode($Input) | Write-Host"

execution time of this oneliner is between 2-3 seconds on virtual machine. Is it because of .NET object is employed? Is there any way to decrease execution time? Have also some C written, lightning fast urldecode.exe utility, but unfortunatelly it does not eat STDIN.

Improve this question
1
  • 2
    I think you need to escape some of those % symbols. Commented Mar 13, 2021 at 23:39

1 Answer 1

Reset to default
4

A note if you're passing the input data as a string literal, as in the sample call in the question (rather than as output from a command):

  • If you're calling from an interactive cmd.exe session, %C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA works as-is - unless any of the tokens between paired % instances happen to be the name of an existing environment variable (which seems unlikely).

  • From a batch file, you'll have to escape the % chars. by doubling them - see this answer; which you can obtain by applying the following PowerShell operation on the original string:

    '%C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA' -replace '%', '%%'

Is it because of .NET object is employed?

  • Yes, powershell.exe, as a .NET-based application requires starting the latter's runtime (CLR), which is nontrivial in terms of performance.

  • Additionally, powershell.exe by default loads the initialization files listed in its $PROFILE variable, which can take additional time.

    • Pass the -NoProfile CLI option to suppress that.

Have also some C written, lightning fast urldecode.exe utility, but unfortunately it does not eat STDIN.

If so, pass the data as an argument, if feasible; e.g.:

urldecode.exe "%C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA"

If the data comes from another command's output, you can use for /f to capture in a variable first, and then pass the latter.

If you do need to call powershell.exe, PowerShell's CLI, after all:

There's not much you can do in terms of optimizing performance:

  • Add -NoProfile, as suggested.
  • Pass the input data as an argument.
  • Avoid unnecessary calls such as Write-Host and rely on PowerShell's implicit output behavior instead.[1]
powershell.exe -NoProfile -c "Add-Type -AssemblyName System.Web;[System.Web.HttpUtility]::UrlDecode('%C4%9B%C5%A1%C4%8D%C5%99%C5%BE%C3%BD%C3%A1%C3%AD%C3%A9%C5%AF%C3%BA')"

[1] Optional reading: How do I output machine-parseable data from a PowerShell CLI call?

Note: The sample commands are assumed to be run from cmd.exe / outside PowerShell.

  • PowerShell's CLI only supports text as output, not also raw byte data.

  • In order to output data for later programmatic processing, you may have to explicitly ensure that what is output is machine-parseable rather than something that is meant for display only.

There are two basic choices:

  • Rely on PowerShell's default output formatting for outputting what are strings (text) to begin with, as well as for numbers - though for fractional and very large or small non-integer numbers additional effort may be required.

  • Explicitly use a structured, text-based data format, such as CSV or Json, to represent complex objects.

Rely on PowerShell's default output formatting:

  • If the output data is itself text (strings), no extra effort is needed. This applies to your case, and therefore simply implicitly outputting the string returned from the [System.Web.HttpUtility]::UrlDecode() call is sufficient:

    # A simple example that outputs a 512-character string; note that 
# printing to the _console_ (terminal) will insert line breaks for 
# *display*, but the string data itself does _not_ contain any 
# (other than a single _trailing one), irrespective of the 
# console window width:
powershell -noprofile -c "'x' * 512"

  • If the output data comprises numbers, you may have to apply explicit, culture-invariant formatting if your code must run with different cultures in effect:

    • True integer types do not require special handling, as their string representation is in effect culture-neutral.

    • However, fractional numbers ([double], [decimal]) do use the current culture's decimal mark, which later processing may not expect:

      # With, say, culture fr-FR (French) in effect, this outputs
# "1,2", i.e. uses a *comma* as the decimal mark.
powershell -noprofile -c "1.2"

# Simple workaround: Let PowerShell *stringify* the number explicitly
# in an *expandable* (interpolating) string, which uses
# the *invariant culture* for formatting, where the decimal
# mark is *always "." (dot).
# The following outputs "1.2", irrespective of what culture is in effect.    
powershell -noprofile -c " $num=1.2; \"$num\" "

    • Finally, very large and very small [double] values can result in exponential notation being output (e.g., 5.1E-07 for 0.00000051); to avoid that, explicit number formatting is required, which can be done via the .ToString() method:

      # The following outputs 0.000051" in all cultures, as intended.
powershell -noprofile -c "$n=0.000051; $n.ToString('F6', [cultureinfo]::InvariantCulture)"

  • More work is needed if you want to output representations of complex objects in machine-parseable form, as discussed in the next section.

    • Relying on PowerShell's default output formatting is not an option in this case, because implicit output and (equivalent explicit Write-Output calls) cause the CLI to apply for-display-only formatting, which is meaningful to the human observer but cannot be robustly parsed.

      # Produces output helpful to the *human observer*, but isn't
# designed for *parsing*.
# `Get-ChildItem` outputs [System.IO.FileSystemInfo] objects.
powershell -noprofile -c "Get-ChildItem /"

    • Note that use of Write-Host is not an alternative: Write-Host fundamentally isn't designed for data output, and the textual representation it creates for complex objects are typically not even meaningful to the human observer - see this answer for more information.

Use a structured, text-based data format, such as CSV or Json:

Note:

  • Hypothetically, the simplest approach is to use the CLI's -OutputFormat Xml option, which serializes the output using the XML-based CLIXML format PowerShell itself uses for remoting and background jobs - see this answer.

  • However, this format is only natively understood by PowerShell itself, and for third-party applications to parse it they'd have to be .NET-based and use the PowerShell SDK.

    • Also, this format is automatically used for both serialization and deserialization if you call another PowerShell instance from PowerShell, with the command specified as a script block ({ ... }) - see this answer. However, there is rarely a need to call the PowerShell CLI from PowerShell itself, and direct invocation of PowerShell code and scripts provides full type fidelity as well as better performance.

  • Finally, note that all serialization formats, including CSV and JSON discussed below, have limits with respect to faithfully representing all aspects of the data, though -OutputFormat Xml comes closest.

PowerShell comes with cmdlets such as ConvertTo-Csv and ConvertTo-Json, which make it easy to convert output to the structured CSV and JSON formats.

Using a Get-Item call to get information about PowerShell's installation directory ($PSHOME) as an example; Get-Item outputs a System.IO.DirectoryInfo instance in this case:

C:\>powershell -noprofile -c "Get-Item $PSHOME | ConvertTo-Csv -NoTypeInformation"
"PSPath","PSParentPath","PSChildName","PSDrive","PSProvider","PSIsContainer","Mode","BaseName","Target","LinkType","Name","FullName","Parent","Exists","Root","Extension","CreationTime","CreationTimeUtc","LastAccessTime","LastAccessTimeUtc","LastWriteTime","LastWriteTimeUtc","Attributes"
"Microsoft.PowerShell.Core\FileSystem::C:\Windows\System32\WindowsPowerShell\v1.0","Microsoft.PowerShell.Core\FileSystem::C:\Windows\System32\WindowsPowerShell","v1.0","C","Microsoft.PowerShell.Core\FileSystem","True","d-----","v1.0","System.Collections.Generic.List`1[System.String]",,"v1.0","C:\Windows\System32\WindowsPowerShell\v1.0","WindowsPowerShell","True","C:\",".0","12/7/2019 4:14:52 AM","12/7/2019 9:14:52 AM","3/14/2021 10:33:10 AM","3/14/2021 2:33:10 PM","11/6/2020 3:52:41 AM","11/6/2020 8:52:41 AM","Directory"

Note: -NoTypeInformation is no longer needed in PowerShell (Core) 7+

C:\>powershell -noprofile -c "Get-Item $PSHOME | ConvertTo-Json -Depth 1"
{
    "Name":  "v1.0",
    "FullName":  "C:\\Windows\\System32\\WindowsPowerShell\\v1.0",
    "Parent":  {
                   "Name":  "WindowsPowerShell",
                   "FullName":  "C:\\Windows\\System32\\WindowsPowerShell",
                   "Parent":  "System32",
                   "Exists":  true,
                   "Root":  "C:\\",
                   "Extension":  "",
                   "CreationTime":  "\/Date(1575710092565)\/",
                   "CreationTimeUtc":  "\/Date(1575710092565)\/",
                   "LastAccessTime":  "\/Date(1615733476841)\/",
                   "LastAccessTimeUtc":  "\/Date(1615733476841)\/",
                   "LastWriteTime":  "\/Date(1575710092565)\/",
                   "LastWriteTimeUtc":  "\/Date(1575710092565)\/",
                   "Attributes":  16
               },
    "Exists":  true
    // ...
}

Since JSON is a hierarchical data format, the serialization depth must be limited with -Depth in order to prevent "runaway" serialization when serializing arbitrary .NET types; this isn't necessary for [pscustomobject] and [hashtable] object graphs composed of primitive .NET types only.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Web server pass POST received data to the STDIN of cmd script, so no need percent doubling in this case. -NoProfile cut execution time by half to 1.2 secs is far enough for me, thanks. Write-Host usage is TRUE necessary in PS oneliners, otherwise PS console cut STDOUT ss64.com/ps/write-host.html
@user2956477, understood re stdin; I've updated the answer to make it clearer that escaping is only needed for literals. Glad to hear that -NoProfile helped. Write-Host is actually not needed with string output, and is in general always the wrong tool for outputting data - please see the bottom section I've added to the answer.
Thanks for your addendum, hopefully I understan it even partialy. Unsing many PS various oneliners in my batch script, so should I replace all trailing Write-Host with Write-Output? Does this reliable avoid unwanted 'console width -1' linewraps?
@user2956477, you don't need explicit Write-Output calls: any result that isn't captured in a variable, sent to another command in a (PowerShell command-internal) pipeline or redirected to a file is implicitly output, as shown in the answer. If you have a console width problem, the implication is that you're mistakenly outputting for-display-only representations of complex objects, and in that case you need to resort to explicit use of structured data formats instead - or explicitly choose to output just one property of such objects that is itself either a string or a number.
Thanks a lot (no irony) you tried to teach me internal PS "output producing" issues. I do not know PS at all, my only 'language' are batch scripts (I am not a programmer). Utilizing PS oneliners founded on internet for tasks out of batch scripts ability. I am not able to discovery which oneliner produce what type of output. I need practical way to ensure the textual output will not be distorted, and Write-Host give me this. Maybe this is not correct way, but it enough and I have no good reason to change it.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.