How to extract specific data from grep command in bash?

Question

I'm trying to write my own script to tell me if I've used more than 500 MiB of my data. I'm using vnstat -d for the information about data usage. vnstat -d Output here

Output should be:

Only from the "Total column"
Only have values greater than 500.

I want only values from the "total"column. My output lists data from all the columns. Better clear from the following:

#!/bin/bash
for i in `vnstat -d | grep -a [0-9] `; //get numerical values in i (-a tag as vnstat outputs in binary)
do 
    NUMBER=$(echo $i | grep -o '[5-9][0-9][0-9]'); //store values >500 in a var called NUMBER
    echo $NUMBER;
done;

I'm a self-learning newb here so please try not to bash (pun) me.

Current output which I'm receiving from above script:

My desired output should be:

654   
923    
967

Please don't post images of text. People here regard that as evil. — John1024
– John1024, Commented Jun 3, 2020 at 23:56
output_cmd | awk -F\| '{print $3} could get your started. Read the awk tutorial . Good luck. — shellter
– shellter, Commented Jun 4, 2020 at 0:09
@Typedragon33 : Why should 600 be excluded? After all, 600 > 500! — user1934428
– user1934428, Commented Jun 4, 2020 at 6:30
Correct, but in your code you do a grep -o '[5-9][0-9][0-9]', which searches all numbers in the line and not only the one in the column titled total. — user1934428
– user1934428, Commented Jun 8, 2020 at 6:09

Ron · Accepted Answer · 2020-06-04 00:29:46Z

1

Simplified:

#/bin/bash
if [[ $(( $(vnstat -d --oneline|cut -d';' -f6|cut -d. -f1|paste -sd '+') )) -ge 500 ]];then
  echo 500 Mb reached
fi

(What the script does, is it takes the specified field from the oneliner CSV-like output from each interface, then cuts the whole numbers and does a SUM of them. And then it compares if that sum is equal or greater than 500. And if it is, then it outputs a message)

Note:

-f6 will parse the "total for today" traffic

you can replace it with:

-f4 = rx for today

-f5 = tx for today

edited Jun 4, 2020 at 0:29

answered Jun 4, 2020 at 0:17

Ron

6,7992 gold badges24 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Typedragon33 Over a year ago

Wow this actually worked! This is genius @Ron. I'm still new and don't understand any of it. But wow man.

Ron Over a year ago

you can adjust the output or echo of course, and you can also check --oneline formats in man vnstat for more output options..

thanasisp · Accepted Answer · 2020-06-04 02:35:31Z

1

You want to parse a pipe delimited table and check only a specific column, there are tools more appropriate than grep for this job, for example you could write a small bash script where you use the cut command to extract the data and process them, or awk.

Here is a solution with awk. We print numbers > 500 of that column, total. Send your command output to

awk -F "|" '($3+0>=500){print $3}'

-F sets the field delimiter to |
$3+0 is used to convert a string starting with a number to that number, so that we can handle it as a number and do the comparison.

Now, if you really want to extract all values having column total > 500 MiB, then the expected output should include all values expressed in GiB, as they are > 1000 MiB, for example the minimum value in your evil screenshot is 0.98 GiB which is 1003 MiB. So we can add this to the first condition.

awk -F "|" '($3 ~ /GiB/ || $3+0>500){print $3}'

Now if you want the output to be only integers in MiB, we can modify it to:

awk -F "|" '($3 ~ /GiB/){$3=1024*$3+0} ($3+0>500){printf "%.0f\n",$3}'

Here we convert all GiB values to MiB, and we do the comparison after that.

edited Jun 4, 2020 at 2:35

answered Jun 4, 2020 at 2:27

thanasisp

6,0053 gold badges18 silver badges32 bronze badges

1 Comment

Typedragon33 Over a year ago

Thanks for such an elaborate and kind explanation! I will take it step by step and come back after I've practiced awk

Shawn · Accepted Answer · 2020-06-04 00:15:30Z

0

I'd use awk. Something like (untested)

vnstat -d | awk '$1 == "estimated" { exit }
                 ($9 == "GiB" && $8 > 0.5) ||
                 ($9 == "MiB" && $8 > 500) { print $8 " " $9 }'

answered Jun 4, 2020 at 0:15

Shawn

53.9k3 gold badges29 silver badges74 bronze badges

Comments

Dat Huynh · Accepted Answer · 2020-06-04 02:27:43Z

0

#!/bin/bash
IFS=$'\n'
for i in `vnstat -d`; do # get each lines
  VALUE=$(echo $i | cut -d\| -f3) # get total value with unit, in case you want to check for GiB values
  NUMBER=$(echo $VALUE | grep -o '[0-9.*]' | cut -d. -f1); # split the string by '|', get the number part, store the integer part into NUMBER
  if [[ $NUMBER -ge 500 && "$VALUE" == *"MiB"* || "$VALUE" == *"GiB"* ]]; then # if the number is greater than or equals to 500 OR it's in GiB
    echo $VALUE; # echo the value
  fi
done

Of course you can strip out the GiB checking if you wanted to.

Edit: Added IFS=$'\n' at the beginning. This allows the for loop to use endline as the delimiter.

edited Jun 4, 2020 at 2:27

answered Jun 3, 2020 at 23:58

Dat Huynh

888 bronze badges

3 Comments

Barmar Over a year ago

That for loop processes each word, not each line.

Typedragon33 Over a year ago

Not working. Here's the output I got. Am I doing something wrong? Here is the output I got textuploader.com/1mghl

Dat Huynh Over a year ago

Oops. I added one more line at the and for the for loop to loop through each line

macaque · Accepted Answer · 2021-04-13 08:38:14Z

vnstat has several options to format the output. You can use vnstat --dumpdb, vnstat --json or vnstat --xml to have well-formatted data that you can then parse more easily (for example with jq if you choose the JSON format).

For example :

vnstat --json | jq '.interfaces[] | select(.id == "eth0") | .traffic | .days[1] | .rx'

will extract the number of kiB received on the interface eth0 yesterday (the day 0 is today, 1 is yesterday, etc)

To have the total rx+tx, you can use

vnstat --json | jq '.interfaces[] | select(.id == "eth0") | .traffic | .total | .rx+.tx'

You can also sum several days, for example today and yesterday :

vnstat --json | jq '.interfaces[] | select(.id == "eth0") | .traffic | [.days[0,1] | .rx+.tx] | add'

And instead of days, you can references "months" or "hours" (for hours, be careful, the id has not the same meaning, it's the reference of the hour).

Collectives™ on Stack Overflow

How to extract specific data from grep command in bash?

5 Answers 5

2 Comments

1 Comment

Comments

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

1 Comment

Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related