Logstash Create an array of values from string

Question

I am having a following set of information in my logs. It is in JSON format.

{   
  "someField":"someValue",   
  "columns":"[colName1, colName2, colName3, ... colNameN]", 
  "someField":"someValue" 
}

I want this to be stored as an array in a same field columns.

The usecase I want to show up is how many users have used the particular column called colName1 and the count of it.

I am using ELK stak 5.x

Joseph M Tsai · Accepted Answer · 2017-05-04 02:50:42Z

2

if your source is directly json format

use the json plugin

filter {
  json {
    source => "message"
  }
}

The array field will auto convert to array

https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html

answered May 4, 2017 at 2:50

Joseph M Tsai

5683 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Vishal Kinjavdekar Over a year ago

I understand that it it will be auto convert to array. but in my case I am using ThreadContext to put the value. and ThreadContext only supports String vales. so although my values looks in an array format, they are actually exported in string format in a log file. So will that affact in this case?

Joseph M Tsai Over a year ago

ThreadContext is the logstash plugin ? if not the json format that you provide should come from message field by default, so i don't know you means ThreadContext

Vishal Kinjavdekar Over a year ago

No its not logstash plugin but it is a library in java that helps me to put values in a field and use it across.

Joseph M Tsai Over a year ago

@VishalKinjavdekar i think your data will still come through the message ,could you check ?

Vishal Kinjavdekar Over a year ago

I checked... In kibana it shows as a text field even after using json plugin

|

Vishal Kinjavdekar · Accepted Answer · 2017-05-08 10:48:27Z

2

I got the solution by referring both the above answers. This is what I did.

To remove [ and ] from the string I used this:

mutate {
        gsub => ["columns", "[\[\]]", ""]
    }

and then to convert it into array:

    mutate {
        split => { "columns" => "," }
    }

answered May 8, 2017 at 10:48

Vishal Kinjavdekar

3211 gold badge5 silver badges14 bronze badges

Comments

Amit Yadav · Accepted Answer · 2017-05-03 16:32:11Z

1

Mutate in output.conf might help you.

Follow this link - https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-replace

Although direct conversion from string to array is not supported. I faced same use case some time back, used merge functionality for that. I would suggest have some dummy field and merge with 'columns' field to create new array field. Hope this helps.

answered May 3, 2017 at 16:32

Amit Yadav

1,0399 silver badges21 bronze badges

Collectives™ on Stack Overflow

Logstash Create an array of values from string

3 Answers 3

9 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

9 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related