Regex not working in curl output

Question

Im using curl to send a POST request in debian linux terminal and its working properly, This is the curl command:

curl --data "ping=8.8.8.8" -s http://www.ipvoid.com/ping/

Now i want to capture the content between the <textarea> tags by executing this command:

curl --data "ping=8.8.8.8" -s http://www.ipvoid.com/ping/ | grep -ioE '<textarea.*>(.*(\n.*)*)<\/textarea>'

But it returns nothing. I tested the regex and it works properly:

regex101.com

Is the problem with the regex or grep syntax?

It works when your input is a multiline string. grep parses input line by line. — Wiktor Stribiżew
– Wiktor Stribiżew, Commented Apr 5, 2018 at 17:18
The xmllint answer is obviously the best, but the perl is: perl -0 -ne 'print for /<textarea.*>([\s\S]*?)<\/textarea>/gi' (Your regex works too, if you make the inner group non-capturing ((?:)) — zzxyz
– zzxyz, Commented Apr 5, 2018 at 17:40
using grep -Pzo "<textarea.*>(.*(\n.*)*)<\/textarea>" but <textarea> tag still exist — CheckYourSec
– CheckYourSec, Commented Apr 5, 2018 at 18:56

RomanPerekhrest · Accepted Answer · 2018-04-05 17:27:08Z

2

Since the result of the crucial HTTP request is HTML document the right way is to apply xml/html parsers.

xmllint is one of such:

curl -d "ping=8.8.8.8" -s http://www.ipvoid.com/ping/ \
| xmllint --html --xpath '//textarea/text()' - 2>/dev/null

The output:

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=61 time=1.12 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=61 time=1.05 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=61 time=1.14 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.052/1.107/1.144/0.039 ms

http://xmlsoft.org/xmllint.html

answered Apr 5, 2018 at 17:27

RomanPerekhrest

93.1k4 gold badges75 silver badges112 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

hanshenrik · Accepted Answer · 2018-04-05 20:25:32Z

0

by default, grep parses the input individually per line, and your textarea has newlines in it, thus your regex doesn't work. but you can (ab)use the --null-data parameter, then it will separate the input by NULL bytes instead of newlines, and since there's no NULL bytes in your textarea, it works!

curl --data "ping=8.8.8.8" -s http://www.ipvoid.com/ping/ | grep -ioE '<textarea.*>(.*(\n.*)*)<\/textarea>' --null-data

(but i recommend using a proper HTML parser instead, the xmllint recommended by @RomanPerekhrest would probably be a better solution, if it's available to you)

answered Apr 5, 2018 at 20:25

hanshenrik

22.4k4 gold badges62 silver badges113 bronze badges

1 Comment

CheckYourSec Over a year ago

This will return the <textarea> too. I just want the content between the tags. That xmllint seems working but i want to do this with grep or perl which is possible, Already done with perl in @zzxyz comment, Now im trying with grep.

Collectives™ on Stack Overflow

Regex not working in curl output

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related