Way to populate Logstash output variable without getting it from an Input?

Question

Is there another way to tell Logstash to supply a value to an output variable without pulling it from a Logstash input? For example, in my case I'd like to create an Elasticsearch index based on a performance run ID (which I'd do from an external script) and then have Logstash send to that. For now I was thinking of creating a tcp input just for receiving perf run info and then have a filter to match on the run id. Seems like a convoluted way to do this though. For example:

input {
    tcp {
        type => "perfinfo"
        port => 8888
    }
}

if [type] == "perfinfo" {
    do some matching to extract the id
}

output {
    elasticsearch { 
        cluster => "mycluster" 
        manage_template => false 
        index => "%{id}-perftest" 
    }
}

I'm not sure if setting manage_template to false would actually be necessary. I've read that it is.

Update

Thanks Nirdesh for that. Using Ruby might be very handy.

While I was waiting I tried using a grok filter like so:

grok {
  match => { "message" => "%{WORD:perftype}-%{POSINT:perfid}" }
}

Which produced this stdout during debugging:

{
   "message" => "awperf-14",
  "@version" => "1",
"@timestamp" => "2014-10-17T20:01:19.758Z",
      "host" => "0:0:0:0:0:0:0:1:33361",
      "type" => "perfinfo",
  "perftype" => "awperf",
    "perfid" => "14"
}

Which I tried creating an index based on this like so:

index => "%{perftype}-%{perfid}"

So when I passed 'awperf-14' to the input, I ended up creating these indexes

%{perftype}-%{perfid}

awperf-14

Which is not what I was expecting. Also, it's the %{perftype}-%{perfid} index that starts to be populated, not awperf-14, the one I actually wanted.

Nirdesh Sharma · Accepted Answer · 2014-10-17 19:33:36Z

4

Yes.

You can add any no. of your own variables either for intermediate result or for permanent using a property called add_field. All most all filters in logstash support this property.

So, for your soluation, you can use a ruby script to find out the id dynamically and store it in a new variable called id, which you can use it in output.

For Example :

input {
   tcp {
       type => "perfinfo"
       port => 8888
   }
}

filter{
  if [type] == "perfinfo" {
    ruby{
         //do some processing
        add_field => { "id" => "Some value" }       
    }         
   }
}

output {
   elasticsearch { 
       cluster => "mycluster" 
       manage_template => false 
       index => "%{id}-perftest" 
  }
}

answered Oct 17, 2014 at 19:33

Nirdesh Sharma

7427 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user4010306 Over a year ago

Hi Nirdesh, see my update above. I've also since tried adding a field and using that field as the variable, but I still have an issue where it creates two indexes. One index with the unevaluated variable name, and the other with the variable value. Not sure why it is doing this.

score 0 · Accepted Answer · 2014-10-21 03:47:17Z

0

I'm not sure I can do what I was trying to do via Logstash. To be a clearer, I simply wanted to change the index based on the performance run ID I'm executing. There's nothing in the data that would have this information (I have to pull it from a DB). So instead of trying to have Logstash listen for a performance run ID, I scripted this externally. The script uses the Elasticsearch API to create a new index, and then does a string replace for the index in the Logstash config file. It then restarts Logstash, which normally happens between performance runs anyway. This approach was much easier to do, and seems cleaner.

edited Oct 21, 2014 at 3:47

answered Oct 20, 2014 at 23:17

user4010306

Collectives™ on Stack Overflow

Way to populate Logstash output variable without getting it from an Input?

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related