Make newly created PowerShell files default to UTF-8 *with BOM* to avoid encoding misinterpretation

[mklement0]

Summary of the new feature

• VSCode creates UTF-8 files without BOM by default.
• This causes Windows PowerShell (but not PowerShell Core) to misinterpret any non-ASCII-range characters, because, in the absence of a BOM, it defaults to the system's legacy "ANSI" code page (e.g., Windows-1252).

It seems that this has been a perennial pain point (whose root cause isn't obvious), as evidenced by, for instance, by #629 or this StackOverflow question.

Making the extension default all new PowerShell files to UTF-8 with BOM solves that problem.

Such files would be both cross-edition and cross-platform compatible (given that PowerShell Core still correctly interprets the BOM, even though it doesn't require it).

Proposed technical implementation details

I don't know how it works in the context of extension-specific settings, but in the general settings.json file you can simply add the following:

"[powershell]": {
  "files.encoding": "utf8bom"
}

[rjmholt]

The current big issue here is that the extension is third-party software in the eyes of both VSCode and PowerShell.

There's no exposed API for it to configure this VSCode setting on installation.

See microsoft/vscode#824.

Anyone seeing issues here, please lend your 👍 to microsoft/vscode#824 and take a look at MicrosoftDocs/PowerShell-Docs#3743 (will update to the doc link when it's merged).

@mklement0 I assume that document prompted this issue?

[mklement0]

Thanks, @rjmholt - it's unfortunate that there's still no API for this (I've since given the linked issue a thumbs-up).

I assume that document prompted this issue?

No, I wasn't aware of that document (thanks for the link). It was my own experience and seeing people run into the problem on SO (Stack Overflow) that prompted me to create this issue.

I've now (hopefully) given it wider exposure with this SO answer.

[rjmholt]

I've now (hopefully) given it wider exposure with this SO answer.

Ah! I'll link to it in the new doc

[rkeithhill]

With the new doc on handling encoding WRT PowerShell and text editors, can we close this?

[mklement0]

I suggest keeping this open with an Resolution-External label (or equivalent) to await being able to implement a proper solution via the future API proposed in microsoft/vscode#824.

[rjmholt]

Doc: https://docs.microsoft.com/en-us/powershell/scripting/components/vscode/understanding-file-encoding?view=powershell-6

[irvnriir]

Windows PowerShell just can't pull a single change of the default . better users would be making files with BOM for no reason ..

[andyleejordan]

@mklement0 nowadays it is possible for the extension to supply a default configuration for powershell files:

"[powershell]": {
    "files.encoding": "utf8bom",
    "files.autoGuessEncoding": true
}

This could go here:

vscode-powershell/package.json

Lines 885 to 890 in 8f7649d

	"configurationDefaults": {
	"[powershell]": {
	"debug.saveBeforeStart": "nonUntitledEditorsInActiveGroup",
	"editor.semanticHighlighting.enabled": false,
	"editor.wordSeparators": "`~!@#$%^&*()=+[{]}\\|;:'\",.<>/?"
	}

I can't think of anything it would break...as you pointed out, PowerShell Core readily accepts UTF8BOM, and it fixes issues with Windows PowerShell. @rjmholt can you think of any reasons not to do this now?