logstash issue with json input file

Question

i have the following json in a file-

{
"foo":"bar",
"spam" : "eggs"
},
{
"css":"ddq",
"eeqw": "fewq"
}

and the following conf file-

input { 
file
{ 
   path => "/opt/logstash-1.4.2/bin/sam.json"
   type => "json"
   codec => json_lines
   start_position =>"beginning"
 }
}
output { stdout {  codec => json  } }

but when i run

./logstash -f sample.conf

i don't get any output in stdout.

but when i don't give json as codec and give type => "core2" then it seems to work. Anyone know how i can fix it to work for json type.

The other issue is it gives me the following output when it does give stdout-

{"message":"{","@version":"1","@timestamp":"2015-07-15T02:02:02.653Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"\"foo\":\"bar\", ","@version":"1","@timestamp":"2015-07-15T02:02:02.654Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"\"spam\" : \"eggs\" ","@version":"1","@timestamp":"2015-07-15T02:02:02.655Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"},","@version":"1","@timestamp":"2015-07-15T02:02:02.655Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"{ ","@version":"1","@timestamp":"2015-07-15T02:02:02.655Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"\"css\":\"ddq\", ","@version":"1","@timestamp":"2015-07-15T02:02:02.656Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"\"eeqw\": \"fewq\"","@version":"1","@timestamp":"2015-07-15T02:02:02.656Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"}","@version":"1","@timestamp":"2015-07-15T02:02:02.656Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}{"message":"","@version":"1","@timestamp":"2015-07-15T02:02:02.656Z","type":"core2","host":"sjagannath","path":"/opt/logstash-1.4.2/bin/sam.json"}

I want to know how it can be parsed the right way with the key value pairs in my input file

Community · Accepted Answer · 2017-05-23 11:50:35Z

3

I found this and edited it to suit your purpose. The following config should do exactly what you want:

input {   
file     {
    codec => multiline
    {
        pattern => "^\}"
        negate => true
        what => previous               
    }
    path => ["/absoute_path/json.json"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
}
}

filter {
mutate    {
    replace => [ "message", "%{message}}" ]
    gsub => [ "message","\n",""]
    gsub => [ "message","},",""]
}
if [message] =~ /^{.*}$/     {
    json { source => message }
}
}

I tried your given json and it results in two events. First with foo = bar and spam = eggs. Second with css = ddq and eeqw = fewq.

edited May 23, 2017 at 11:50

CommunityBot

11 silver badge

answered Jul 15, 2015 at 11:46

hurb

2,2273 gold badges20 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

sharat Over a year ago

i works partially. But i can read only the first json data and not the 2nd (with css). Is there something missing?

hurb Over a year ago

Might be a missing newline at the end of your json file. Logstash won't read the last line until it is finished with a newline.

sharat Over a year ago

ok.Suppose i don't have the comma separator. how would the filter look like? i tried gsub => [ "message","}",""] but it didn't work

hurb Over a year ago

Try this as second gsub: gsub => [ "message","}{","{"] It should properly read your data without a comma separator.

Community · Accepted Answer · 2017-05-23 11:59:25Z

0

As of my understanding you want to put your complete son document on one line if you want to use the json_lines codec:

{"foo":"bar","spam" : "eggs"}
{"css":"ddq","eeqw": "fewq"}

In your case you have a problem with the structure since you also have a ',' between the son objects. Not the most easy way to handle it. SO if possible change the source to my example. If that is not possible the multiline approach might help you. Check this for reference: input json to logstash - config issues?

edited May 23, 2017 at 11:59

CommunityBot

11 silver badge

answered Jul 15, 2015 at 5:30

Jettro Coenradie

4,73325 silver badges31 bronze badges

1 Comment

hurb Over a year ago

In fact, changing the source might be an approach. However, if that is not possible, see my answer.

Collectives™ on Stack Overflow

logstash issue with json input file

2 Answers 2

4 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related