bash script with string extract sometimes extracts wrong

Question

To preface this: the last time I wrote "code" was about 20 years ago. It was some QBasic script in school. My ISP heavily throttles my Netflix traffic and is all around kind of douchey, I don't get the full bandwidth (I know it's "best effort", but still). So I bought a RPi3, went to the Google and wildly copied code from other people.

Now I have a Python script (that should be well known) speedtest.py, and I got a bash script, that extracts the relevant things and writes it to a MySQL Database.

# temp file holding output to snip it
user=$USER
if test -z $user; then
  user=$USERNAME
fi
log=/tmp/$user/speedtest-mysql.log

# Local functions
str_extract() {
 pattern=$1
 # Extract
 res=`grep "$pattern" $log | sed "s/$pattern//g"`
 # Drop trailing ...
 res=`echo $res | sed 's/[.][.][.]//g'`
 # Trim
 res=`echo $res | sed 's/^ *//g' | sed 's/ *$//g'`
 echo $res
}

mkdir -p `dirname $log`

# run speedtest
/usr/local/bin/speedtest --share > $log

# parse the output
from=`str_extract "Testing from "`
from_ip=`echo $from | sed 's/.*(//g' | sed 's/).*//g'`
from=`echo $from | sed 's/ (.*//g'`

server=`str_extract "Hosted by "`
server_ping=`echo $server | sed 's/.*: //g'`
server=`echo $server | sed 's/: .*//g'`
server_dist=`echo $server | sed 's/.*\\[//g' | sed 's/\\].*//g'`
server=`echo $server | sed 's/ \\[.*//g'`

download=`str_extract "Download: "`
upload=`str_extract "Upload: "`
share_url=`str_extract "Share results: "`

#create timestamp for database
timestamp=`date +"%Y-%m-%d %H:%M:%S"`

# Send to MySQL

value1=`echo $server_ping | cut -d" " -f1`
value2=`echo $download | cut -d" " -f1`
value3=`echo $upload | cut -d" " -f1` 

sql="INSERT INTO $db_table (timestamp,ping,download,upload,server,share_url) VALUES ('$timestamp','$server_ping','$download','$upload','$server','$share_url');"

echo "$sql" | mysql -u$db_user -p$db_passwd -h$db_host $db_name

Problem now is: sometimes I get an output from the speedtest with 95.53 MBit/s Download and 9.93 MBit/s Upload, but my Database gets filled with an upload value of 0.90. The relevant output of speedtest.py that is written to the logfile:

Retrieving speedtest.net configuration...
Testing from UPC Austria (80.110.109.207)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Timewarp IT Consulting GmbH (Vienna) [2.79 km]: 32.403 ms
Testing download speed................................................................................
Download: 17.46 Mbit/s
Testing upload speed................................................................................................
Upload: 2.28 Mbit/s
Share results: http://www.speedtest.net/result/7075502570.png

as said: My last voyage to coding was back in the 90s, and my google-fu is failing to find me an answer to this problem. Can somebody help?

You could remove python and raspberry-pi tags, or also add the relevant parts of the script. Most likely it is some conversion problem. — CristiFati
– CristiFati, Commented Feb 20, 2018 at 18:24
Lots of missing quotes here. Consider running this through shellcheck.net and fixing what it finds. Building a minimal reproducible example with the shortest possible code that demonstrates a problem would also be helpful -- for instance, if instead of running /usr/local/bin/speedtest you just wrote a hardcoded string, then someone who didn't have speedtest installed could run the script themselves to see the problem, and thus be able to test their (or someone else's) proposed fix. — Charles Duffy
– Charles Duffy, Commented Feb 20, 2018 at 18:40
/usr/local/bin/speedtest is actually a python script from github.com/sivel/speedtest-cli/blob/master/speedtest.py I will edit my code above to be better readable and use quotes. — Johannes
– Johannes, Commented Feb 20, 2018 at 18:51

ghoti · Accepted Answer · 2018-02-20 19:01:29Z

As Charles suggested in comments, there are many things that could be improved with this script. While shellcheck.net will help you identify things that are wrong with your script, it won't help much with strategy.

It seems to me that you could eliminate many hassles with something like this:

#!/usr/bin/env bash

IFS=,

a=( $(/usr/local/bin/speedtest.py --csv) )

fmt="INSERT INTO %s (timestamp,ping,download,upload,server)
  VALUES ('%s',%.2f,%.2f,%.2f,'%s')"

printf -v sql "$fmt" "${a[4]}" "${a[6]}" "${a[7]}" "${a[8]}" "${a[3]}"

mysql -u"$db_user" -p"$db_passwd" -h"$db_host" -e "$sql" "$db_name"

This solution assumes that speedtest.py will never produce output that embeds a comma within a quoted string. No idea if the "Sponsor" field ever contains commas, you could add error checking around that if you like, perhaps even just by counting fields.

The CSV output is loaded into the fields of an array, $a[], which printf accesses in populating the $sql variable.

You would of course adjust the format to suit your requirements. And the script above uses a few more variables than it strictly needs to, for the sake of readability.

Note that --csv and --share options appear to be mutually exclusive.

Collectives™ on Stack Overflow

bash script with string extract sometimes extracts wrong

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related