Ruby script for json formatter [duplicate]

Question

I am making json formater for sonarqube, here is my script:

require 'json'

tr_report = File.open('./tes.json').read
tr_report.gsub!(/\r\n?/, "\n")

sq_generic_format = {'issues' => []}
sq_issue_format = {
  'engineId' => '', # CONST='brakeman'
  'ruleId' => '', #[check_name (warning_code)] warning_type [confidence]
  'severity':'MAJOR', # MAJOR
  'type':'VULNERABILITY', # CONST='VULNERABILITY'
  'primaryLocation' => {},
  'effortMinutes' => 0, #CONST=0
}

primary_location_format = {
  'message' => '', # message + CONST='\nCode:' + code + CONST='\nUser Input:' + user_input + CONST='\nLink: ' + link
  'filePath' => '', # file
  'textRange' => {}
}

text_range_format = {
  'startLine' => 1,# line
  'endLine' => 1,# line
  'startColumn' => 0,
  'endColumn' => 1
}

issues = []

tr_report.each_line do |line|
  tr_data = JSON.parse(line)
  # puts tr_data
  # puts parsed["SourceMetadata"]["Data"]["Filesystem"]["file"]
  issue = sq_issue_format
  issue['engineId'] = 'trufflehog'
  issue['ruleId'] = 'Sensitive Data Exposure - %s' %[tr_data['Raw']]
  issue['severity'] = 'MAJOR' # MAJOR
  issue['type'] = 'VULNERABILITY' # CONST='VULNERABILITY'
  issue['effortMinutes'] = 0

  issue['primaryLocation'] = {}
  # filling up nested data lvl1 ^
  primary_location = primary_location_format
  primary_location['message'] = 'Sensitive Data Exposure'
  primary_location['filePath'] = tr_data["SourceMetadata"]["Data"]["Filesystem"]["file"] # file

  primary_location['textRange'] = {}
  # filling up nested data lvl2 ^
  text_range = text_range_format
  # text_range['startLine'] = w['line']
  # text_range['endLine'] = w['line']

  # sticking all together
  primary_location['textRange'] = text_range
  issue['primaryLocation'] = primary_location

  issues.append(issue)
end
# puts issues
sq_generic_format['issues'] = issues
puts JSON.dump(sq_generic_format)
File.write('./trufflehog-sq-report.json', JSON.dump(sq_generic_format))

and here is my jsonline tes.json:

{"SourceMetadata":{"Data":{"Filesystem":{"file":"../ruby/railsgoat/dependency-check-report.html"}}},"SourceID":15,"SourceType":15,"SourceName":"trufflehog - filesystem","DetectorType":9,"DetectorName":"Gitlab","Verified":false,"Raw":"vulnerable-to-driveby-","Redacted":"","ExtraData":null,"StructuredData":null}
{"SourceMetadata":{"Data":{"Filesystem":{"file":"../ruby/railsgoat/dependency-check-report.html"}}},"SourceID":15,"SourceType":15,"SourceName":"trufflehog - filesystem","DetectorType":800,"DetectorName":"Atera","Verified":false,"Raw":"39a6bda16ef9583fba2696cc3efde0da","Redacted":"","ExtraData":null,"StructuredData":null}

But everytime I try run it, I always got the first line of the parse, I cant get the next line and make the result redundace. how to capture my parse for the next line? and not just the first line.

Also i make a simple script to parse the jsonl, and it successfully like my expected. here is the script:

require 'json'

text=File.open('tes.json').read
text.gsub!(/\r\n?/, "\n")
text.each_line do |line|

  parsed = JSON.parse(line)
  puts parsed["Raw"]
end

result:

vulnerable-to-driveby-
39a6bda16ef9583fba2696cc3efde0da

The current result: its parse just only the first line, expected result: I got all of the parse properly. My expected result for my formatter script:

 {"issues":[{"engineId":"trufflehog","ruleId":"Sensitive Data Exposure - vulnerable-to-driveby-","severity":"MAJOR","type":"VULNERABILITY","primaryLocation":{"message":"Sensitive Data Exposure","filePath":"../ruby/railsgoat/dependency-check-report.html","textRange":{"startLine":1,"endLine":1,"startColumn":0,"endColumn":1}},"effortMinutes":0,"severity":"MAJOR","type":"VULNERABILITY"},{"engineId":"trufflehog","ruleId":"Sensitive Data Exposure - 39a6bda16ef9583fba2696cc3efde0da","severity":"MAJOR","type":"VULNERABILITY","primaryLocation":{"message":"Sensitive Data Exposure","filePath":"../ruby/railsgoat/dependency-check-report.html","textRange":{"startLine":1,"endLine":1,"startColumn":0,"endColumn":1}},"effortMinutes":0,"severity":"MAJOR","type":"VULNERABILITY"}]}

and here what i got right now:

{"issues":[{"engineId":"trufflehog","ruleId":"Sensitive Data Exposure - 39a6bda16ef9583fba2696cc3efde0da","severity":"MAJOR","type":"VULNERABILITY","primaryLocation":{"message":"Sensitive Data Exposure","filePath":"../ruby/railsgoat/dependency-check-report.html","textRange":{"startLine":1,"endLine":1,"startColumn":0,"endColumn":1}},"effortMinutes":0,"severity":"MAJOR","type":"VULNERABILITY"},{"engineId":"trufflehog","ruleId":"Sensitive Data Exposure - 39a6bda16ef9583fba2696cc3efde0da","severity":"MAJOR","type":"VULNERABILITY","primaryLocation":{"message":"Sensitive Data Exposure","filePath":"../ruby/railsgoat/dependency-check-report.html","textRange":{"startLine":1,"endLine":1,"startColumn":0,"endColumn":1}},"effortMinutes":0,"severity":"MAJOR","type":"VULNERABILITY"}]}

PS: see the ruleId for the difference.

After fixing your question formatting so tes.json shows properly (i.e. as code block, not inline code), your code works; I get two elements in issues. This assumes what was in the source of your question was accurate, in which case you have a JSONL file, not a JSON file. Please re-check if the question contains the correct data (thus, it is important to get the formatting correctly), and what exact result you are expecting, and what exact result you are getting. — Amadan
– Amadan, Commented Nov 4, 2022 at 2:15
Yes its invalid as a json file, thank you for the reference to the jsonl. Actually i have a script to parse it and get the all of data without redundance. the problem is when I try to serialize it using above script. It always just parse for the first line. I think its because lack of loop logic so its not iterate the next line. @Amadan — nyoba hack
– nyoba hack, Commented Nov 4, 2022 at 2:37
As I said, I am getting all the results using the above code and data: { "issues": [ { "engineId": ... }, { "engineId": ... } ] }. Please run the code, and paste the actual results into your question, so we can examine them together, then post what you actually expect (as actual JSON, not as a description). — Amadan
– Amadan, Commented Nov 4, 2022 at 2:39
I have edited the question. and add the expected result. thank you in advance for your help. — nyoba hack
– nyoba hack, Commented Nov 4, 2022 at 3:01
Ah, I see. The reason you are getting the same row multiple times is you keep changing one object (sq_issue_format) and pushing that same object into your array multiple times, overwriting its contents in each loop iteration. Replace issue = sq_issue_format with issue = sq_issue_format.dup. Someone should be coming along soon with a good duplicate to close (I know they exist, but can't find one right now). Think "I met Jane in green shirt, and Jane in red shirt, why do all the Janes have the same face?" because it's the same person, who changed clothes :) — Amadan
– Amadan, Commented Nov 4, 2022 at 3:07

Travis · Accepted Answer · 2022-11-03 18:11:25Z

1

First, I ran your json through a formatter and it was reported as invalid. If you're going to have multiple objects you should use an array. So I've adjusted it to be: [{...},{...}]. (this is because JSON expects there to only be 1 root eleemnt.)

I think it's easiest to then say, you're doing the work that the JSON.parser is already meant to do. You can iterate through the object directly off the parser: JSON.parse(File.read("/tmp/tes.json")).map{ |obj| obj["Raw"] }

This gives me the results of => ["vulnerable-to-driveby-", "39a6bda16ef9583fba2696cc3efde0da"]

answered Nov 3, 2022 at 18:11

Travis

3,1984 gold badges24 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Ruby script for json formatter [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related