Windows (NTFS): Inconsistent support for the long-path prefix (\\?\) in filesystem paths

[mklement0]

Note: This is a generalization of #4439. @SteveL-MSFT, if you agree, please close the latter.

Prefixing full, normalized, native filesystem paths with \\?\ allows targeting filesystem items whose path is longer than the legacy limit of 259 characters.

Update:

• Prefix \\?\ isn't needed in PS Core (in .NET Core altogether), so long paths just work as-is, even if not enabled system-wide.
• However, old code may still use it, and even new code may have to, when creating cross-edition scripts.
• Additionally, and separately, \\?\ is useful for targeting files or directories with irregular names, such as ones with trailing spaces, so you can remove them with Remove-Item, for instance.

As an aside: In Windows 10 you can now opt in system-wide to support long paths, but individual applications must also opt in (PowerShell indirectly does) - see https://blogs.msdn.microsoft.com/jeremykuhne/2016/07/30/net-4-6-2-and-long-paths-on-windows-10/

As of PowerShell Core 7.3.0-preview.2, the inconsistent support for \\?\ is as follows:

• With the -Path parameter, \\?\ isn't supported at all , so wildcard expressions cannot be used.

  • Get-Item and Get-ChildItem output nothing, and Remove-Item is a quiet no-op (silent failure) too.

• With the -LiteralPath parameter in combination with \\?\:

  • Only file paths can be targeted.
  • Directory paths exhibit the following behavior:
    • Get-Item and Remove-Item complain about not finding the path.
      • Adding -Force makes Get-Item output a broken DirectoryInfo instance.
      • Using Remove-Item without -Recurse on a nonempty directory presents the usual confirmation prompt (implying the ability to recognize the path as existent), but then fails on confirmation.
    • Get-ChildItem reports the root directory's content instead(!)

Additionally, \\?\ paths do not work in the following cases:

• With > / >>, which implicitly behave like Out-File -Path - see Some scripts / files with wildcard metacharacter [ in the file name or path cannot be invoked or redirected to #4726

• In the same vein, invocation of an executable - whether with or without & - is broken (possibly related to Some scripts / files with wildcard metacharacter [ in the file name or path cannot be invoked or redirected to #4726 as well). (no longer a problem as of PowerShell Core 7.3.0-preview.2)

• Start-Process (no longer a problem as of PowerShell Core 7.3.0-preview.2)

• Set-Location (not even with -LiteralPath)

Note:

• Some of these problems are regressions from Windows PowerShell, where only the invocation / Start-Process tests fail and > only with a new file.

• I haven't looked into whether invoking an executable with an overly long path is supported in principle by the underlying APIs.

Steps to reproduce

Run the following from a Pester test script (*.Tests.ps1) on Windows:

Describe "Support for long paths, via \\?\" {
  BeforeAll {
    Push-Location (Convert-Path TestDrive:\)
    $dirPath = $PWD.ProviderPath
    $PrefixedDir = "\\?\$dirPath"
    $fileName = ('a' * 248 + '.cmd')
    $fileNameAlt = ('b' * 248)
    $PrefixedFullName = "$PrefixedDir\$fileName"
    $PrefixedFullNameAlt = "$PrefixedDir\$fileNameAlt"
    # Create the file with the overly long path using .NET,
    # to avoid issues with New-Item
    [IO.File]::WriteAllText($PrefixedFullName, '')
  }
  It "Get-ChildItem -LiteralPath" {
    Get-ChildItem -LiteralPath $PrefixedFullName | % FullName | Should -Be $PrefixedFullName
  }
  It "Get-ChildItem -Path" {
    Get-ChildItem -Path $PrefixedFullName | % FullName | Should -Be $PrefixedFullName
  }
  It "Remove-Item -LiteralPath" {
    { Remove-Item -LiteralPath $PrefixedFullName } | Should -Not -Throw
    # Recreate the file.
    [IO.File]::WriteAllText($PrefixedFullName, '')
  }
  It "Remove-Item -Path" {
    { Remove-Item -Path $PrefixedFullName } | Should -Not -Throw
    # Recreate the file, if necessary
    [IO.File]::WriteAllText($PrefixedFullName, '')
  }
  It "> with new file" {
    { '' >  $PrefixedFullNameAlt } | Should -Not -Throw
  }
  It "> / >> with existing file." {
    { '@echo Hi.' >  $PrefixedFullName } | Should -Not -Throw
    { 'REM ' >> $PrefixedFullName } | Should -Not -Throw
  }
  It "Invocation / &" {
    & $PrefixedFullName | Should -Be 'Hi.'
  }
  It "Start-Process" {
    Start-Process -FilePath $PrefixedFullName
  }
  It "New-Item" {
    { New-Item -Force -Type File $PrefixedFullName } | Should -Not -Throw
  }
  It "Set-Location -LiteralPath" {
    { Set-Location -EA Stop -LiteralPath $PrefixedDir } | Should -Not -Throw
  }
  It "Set-Location -Path" {
    { Set-Location -EA Stop -Path $PrefixedDir } | Should -Not -Throw
  }
  AfterAll {
    # Use .NET to remove the overly long path, so that Pester itself doesn't 
    # fail on trying to remove the dir. underlying TestDrive:
    [IO.File]::Delete($PrefixedFullName)
    [IO.File]::Delete($PrefixedFullNameAlt)
    Pop-Location
  }
}

Expected behavior

All tests should pass.

Actual behavior

All tests but the first one fail, with various error messages.

Environment data

PowerShell Core 7.0.0-preview.4

[iSazonov]

LocalGlobbing code is very sensitive and it seems it is not covered by tests explicitly - I think it is main stopper of a progress here.

[mklement0]

As of PowerShell Core 7.0.0-rc.2 Get-Item -LiteralPath \\?\C:\Windows is now broken too (complains about not finding the path; Get-ChildItem -LiteralPath \\?\C:\Windows still works).

I've noticed that the \\?\ prefix is never needed in PS Core. Still, people may use it in cross-edition scripts.

[iSazonov]

I've noticed that the \?\ prefix is never needed in PS Core.

Long paths is Windows feature, not PS Core. It seems we have a test for long path support.

[mklement0]

Yes, it only applies on Windows, as hopefully clearly implied by the issue title. PS Core should support the long-path prefix there too, and support has degraded from Windows PowerShell, and even further since v7.0.0-preview.4

What I was trying to say: In PowerShell Core on Windows, you can seemingly access paths longer than 259 chars. even without the \\?\ prefix, irrespective of whether long-path support is turned on system-wide.

[mklement0]

It seems we have a test for long path support.

You can run the tests from the initial post; for me, 7 out of the 11 tests fail as of v7.0.0-rc.2

[iSazonov]

What I was trying to say: In PowerShell Core on Windows, you can seemingly access paths longer than 259 chars. even without the \?\ prefix, irrespective of whether long-path support is turned on system-wide.

I don't understand how PowerShell Core can work with long paths if the feature is not enabled on Windows 10.

[mklement0]

That is a fair question, @iSazonov - I haven't looked into the why and how, but you can try it yourself:

First, open an elevated session and run the following, to turn system-wide support for long paths off:

Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem LongPathsEnabled 0

Then, open a regular session and run the following:

Set-Location; 'hi' > ('x'*(259 - "$pwd".length)); (Get-Item x*).FullName.Length; Remove-Item x*

You'll get no error message and 260 as the output, which is the longer-than-259-chars. path of the temporarily created file.

Running the same code in Windows PowerShell fails.

Perhaps \\?\ is being applied behind the scenes?

[iSazonov]

@mklement0 Please share your Windows 10 version. I guess latest versions can have the feature enabled by default.

[mklement0]

It's Microsoft Windows 10 Pro (64-bit; Version 1909, OS Build: 18363.592), but I don't think that is the issue, because the very same code also works on Windows 7 (where system-wide support for long paths didn't even exist).

[iSazonov]

@mklement0 If you replace 259 in your test with more large value (what about 400?) you get an error.

[mklement0]

That is an unrelated constraint: even in long paths the individual components must not be longer than 255 chars., as far as I know. You'd have to design the test differently to get longer paths.

[iSazonov]

Ah :-) my bad.
Get-Item really doesn't process long paths - it split path and works with leaf (the individual component < 256) but returns FileInfo object with FullName which is > 255 (266 chars). Perhaps just the fact confuses us.
Then I tried to Set-Location and create subdirectory but get error on Set-Location. So long paths do not works.