Trying to load in this json file in ruby. Json Parser Error

Question

I am basically trying to load this json file so that I can loop thorough the versions array, but I am getting the following JSON::ParserError.

{
    "key1":  "value",
    "key1":  "value",
    "others_legacy_software":  [
                                   "value",
                                   "value",
                                   "value"
                               ],
    " key1 ":  "...",
    " key1 ":  "...",
    " key1 ":  "...",
    " needed_additional_dataversions":  [
                                            "2020",
                                            "2019",
                                            "2018",
                                            "2017"
                                        ],
    " blahblahblah ":  " "
}

I have tried the following:

file = File.open "releaseinfo.json"
data_versions = JSON.load file
file.close

# I want to be able to do the following:

data_versions['needed_additional_dataversions']

# and then loop over the values for some functions

But I am getting the following error

JSON::ParserError
-----------------
795: unexpected token at '��{'

What unexpected token am I seeing? And how can I solve this problem? Been stuck on this for a couple of hours now

EDIT:

  function gettagsfromdynamo() {
    $table_name = "PC_Deployments"
    $dbkey = '{"ReleaseId":{"AttributeValueList":[ {"N":"1"} ],"ComparisonOperator": "EQ"}}' | ConvertTo-Json -Compress
    $dbvalue = aws dynamodb query --table-name $table_name --key-conditions $dbkey --region $region
    $latest_tags = $dbvalue | ConvertFrom-Json
    $latest_tags
  }
 
  $db_tags = gettagsfromdynamo
  
  $db_tags.Items.Tags.S | Out-File -FilePath releaseinfo.json

Thats my bad. I was changing up the keys to hide the actual data and messed up. That is my mistake here when posting. That isn't the problem. I edited the question — YANS
– YANS, Commented Jan 11, 2021 at 20:49
When I see an error message like unexpected token at '��{' then I am pretty sure that your JSON file is not UTF-8 encoded but was saved in any other encoding. Where did you get that JSON from? How was it created? — spickermann
– spickermann, Commented Jan 11, 2021 at 21:06
The json is returned by Dynamo query and then converted to Powershell object using ConvertFrom-Json (powershell) and then reconverted to Json using ConvertTo-Json and then outputted to a file. — YANS
– YANS, Commented Jan 11, 2021 at 21:21

Schwern · Accepted Answer · 2021-01-11 21:55:01Z

1

The json is returned by Dynamo query and then converted to Powershell object using ConvertFrom-Json (powershell) and then reconverted to Json using ConvertTo-Json and then outputted to a file. – YANS 27 mins ago

Someone set up us the BOM.

My best guess is something in all that converting added a byte-order mark (BOM); a few bytes at the beginning of the file which says how to read it. Most everything now uses UTF-8 and a BOM is rarely needed, but Windows was very late to the UTF-8 party.

If your reader is not expecting a BOM, and most things do not, it will read the BOM as garbage.

Look at the file with a hex editor to find out what the first few bytes of the file really are.

answered Jan 11, 2021 at 21:55

Schwern

167k28 gold badges225 silver badges370 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

YANS Over a year ago

When I use a hex editor and open the .json file, the first two hex numbers before the curly brace (7B) are FF FE, and every letter has a 00 (period) in between. Seems like there is something wrong here

YANS Over a year ago

If it helps, here is the code that generates the json file, a powershell script. I edited my post to add that

Schwern Over a year ago

@YANS FF FE is the UTF-16 BOM read backwards. Make your time.

YANS Over a year ago

So what is the usual way to bypass this. I can open the file using UTF-16 BOM encoding?

Schwern Over a year ago

@YANS "every letter has a 00 (period) in between" This is because it is UTF-16, every character is two bytes. Most characters only need one, so the other will be 0. Ruby reads UTF-8 by default, you have to tell it to use UTF-16 encoding. File.new takes the same options as IO.new.

|

Collectives™ on Stack Overflow

Trying to load in this json file in ruby. Json Parser Error

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related