Powershell: "Unexpected token " error with hebrew letters in string variable [duplicate]

I am using the current VSCode (1.50.1) on Windows10, with the Powershell Integrated Console:

• =====> PowerShell Integrated Console v2020.6.0 <=====

I am unable to reference a file path that has Hebrew letters in it. Why?

With only English in the path, my code works fine:

$DestinationDirMP3 = 'C:\data\personal\hinative-mp3'

But if I copy the path from Explorer and there are Hebrew letters in it,

$DestinationDirMP3 = 'C:\data\personal\עברית\cardbuilding\audio-files\hinative'

I get the "Unexpected token" error shown below:

Error:

At C:\develop\utils\powershell\aac-to-mp3-converter.ps1:19 char:44
+ ... MP3 = 'C:\data\personal\עברית\cardbuilding\audio-files\hinative'
+                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Unexpected token 'רית\cardbuilding\audio-files\hinative'

What I have tried:

• When I use single quotes around teh string, as shown above, I get the error as shown above.
• When I use double quotes as shown above, the error is NOT thrown, but the value in the variable is trashed (in the same way shown in the error), and downstream code uses the trashed value instead of the desired value.

What is the secret sauce I am missing?

(I should add that the file names that flow in and out of this script are full of Hebrew letters, and that all works fine. So the presence of Hebrew generally is not the issue. It seems to be specific to storing Hebrew letters in this variable that is the problem."

SOLUTION

Add the following to your settings.json file (from the command palette (Ctrl+Shift+P, type settings and select Preferences: Open Settings (JSON)):

"[powershell]": {
  "files.encoding": "utf8bom"
}

After this is done, restart VSCode. Now copy and paste your code into a NEW .ps1 file in VSCode and save it. This NEW .ps1 will have the BOM (the BOM does not seem to get added on saving existing .ps1 files). In my case, I did these steps, deleted the original .ps1 and copied my newly created .ps1 to the original .ps1 filename in order to commit to a git repo. (Perhaps I could have saved the new file over the original, but I wanted to see side by side with and without the BOM using a hex editor.)

I should further add that when I take the problem ps1 file (e.g. the file without the BOM), and simply save it under a new name, my observation is that VSCode does NOT add the BOM. The only workflow I have found to get VSCode to add the BOM is to actually open a NEW empty file (ctrl-N), paste the code into it, and save it.

More details are here.