Logstash/Grok: Read substring from field using regex

Question

I'm trying to extract a substring from my request_uri field in logstash. Grok splits my apace access-log line into several field (is already working) so I get the request_uri in its own field. Now I want to get the root context of the uri.

/en/some/stuff
/ApplicationName/some/path
/fr/some/french/stuff

But I don#t know how to store en, ApplicationName, fr in its own field (additional to the others). I'm thinking something like this might work.

grok {
            pattern => "\"%{GREEDYDATA:domain}\" - %{IP:client_ip} \[%{GREEDYDATA:log_timestamp}\] \"%{WORD:method}\" \"%{GREEDYDATA:request_uri}\" - \"%{GREEDYDATA:query_string}\" - \"%{GREEDYDATA:protocol}\" - %{NUMBER:http_statuscode} %{NUMBER:bytes} \"%{GREEDYDATA:user_agent}\" %{NUMBER:seconds} %{NUMBER:milliseconds} \"%{GREEDYDATA:server_node}\""
            match => [ "new_context_field", "SOME-REGEX fo parse request_uri" ]
        }

Can you give me a hint?

Which Logstash version is this? I've never heard of a pattern option to the grok filter. — Magnus Bäck
– Magnus Bäck, Commented Jan 16, 2015 at 10:52

Sebastian Sommerfeld · Accepted Answer · 2015-01-19 12:51:35Z

3

Thanks for your help. Solved it with this grok config which is pretty similar to your suggestion.

grok {
    patterns_dir => "/path/to/elk-stack/logstash-1.4.2/bin/custom_patterns"

    match => [ "message", "\"%{GREEDYDATA:domain}\" - %{IP:client_ip} \[%{GREEDYDATA:log_timestamp}\] \"%{WORD:method}\" \"%{GREEDYDATA:request_uri}\" - \"%{GREEDYDATA:query_string}\" - \"%{GREEDYDATA:protocol}\" - %{NUMBER:http_statuscode} %{NUMBER:bytes} \"%{GREEDYDATA:user_agent}\" %{NUMBER:seconds} %{NUMBER:milliseconds} \"%{GREEDYDATA:server_node}\""]
    match => [ "request_uri", "%{CONTEXTFROMURI:context}" ]

    break_on_match => false
}

To use multiple matches in a single grok block make sure to include break_on_match => false. Otherwise the second match is skipped if first one is successful.

edited Jan 19, 2015 at 12:51

answered Jan 16, 2015 at 12:27

Sebastian Sommerfeld

4071 gold badge6 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Magnus Bäck · Accepted Answer · 2015-01-16 11:01:30Z

2

Your grok filter should actually look like this:

grok {
  match => [
    "message",
    "\"%{GREEDYDATA:domain}\" - %{IP:client_ip} \[%{GREEDYDATA:log_timestamp}\] \"%{WORD:method}\" \"%{GREEDYDATA:request_uri}\" - \"%{GREEDYDATA:query_string}\" - \"%{GREEDYDATA:protocol}\" - %{NUMBER:http_statuscode} %{NUMBER:bytes} \"%{GREEDYDATA:user_agent}\" %{NUMBER:seconds} %{NUMBER:milliseconds} \"%{GREEDYDATA:server_node}\""
  ]
}

Then, use a second grok filter after the one that matches against the whole log message in the 'message' field:

grok {
  match => ["request_uri", "/(?<context>[^/]+)"]
}

answered Jan 16, 2015 at 11:01

Magnus Bäck

11.6k3 gold badges49 silver badges62 bronze badges

Collectives™ on Stack Overflow

Logstash/Grok: Read substring from field using regex

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related