Log from running script sometimes appears as if it is HTML

Question

I have a script that starts either at 00:10 or on reboot, currently running on my NAS. To make sure it was doing what I wanted I have been logging the output. I've tried various methods of writing to the log i.e the normal >> and currently 2>&1 | tee -a whichever the result is the same at some point when I expect the IP to be reported into the log it is shown as if it is HTML Updated IP is <!DOCTYPE html> Currently the script is running with set -x however the result is the same if that is on or off. Sometimes it will go for many hours before corrupting and sometimes it will recover. It always seems to be showing an IP address that does it, I've tried with the address separated by periods and as just digits

The whole script is

 #!/bin/bash

#Uncomment the three below if debugging is needed
set -x
#set -v
#trap "set +x; sleep .1; set -x" DEBUG

#Set some variables
scripthome="/var/services/homes/admin/script"
logfile="/var/services/homes/admin/script/logs/ddns.log"
logfile2="/var/services/homes/admin/script/logs/ddns2.log"
bakfile="/var/services/homes/admin/script/logs/ddns.bak"
bakfile2="/var/services/homes/admin/script/logs/ddns2.bak"
waittime="1800" # time between updates in seconds
now=$(date +%s) #current time in seconds
ipadd="192.168.0.100" #just give it a valid address for now
ipadd2="192.168.0.100" #just give it a valid address for now
ipad="192.168.0.100" #getting odd results in the log so going to have two variable, one of which does not have the . removed
ipad2="192.168.0.100" #getting odd results in the log so going to have two variable, one of which does not have the . removed
runagain=$((now + waittime))
username="myusername"
password="mypassword"
#starttime=$(date +%H:%M)
endtime="23:00"
cd $scripthome || end
#Start & update the current IP
ipadd=$(curl ifconfig.co) #can also use curl wtfismyip.com/text
ipadd2=$ipadd
#remove the periods, easier to compare as an interger
ipad=$ipadd
ipad2=$ipadd2
ipadd="${ipadd//./}"
ipadd2="${ipadd2//./}"
counter=0
updates=0

mv $logfile $bakfile
mv $logfile2 $bakfile2
touch $logfile
touch $logfile2

{
date
echo
echo "next running intial curl, check logfile2"
echo
} 2>&1   |  tee -a $logfile

echo url="https://api.dynu.com/nic/update?username=$username&password=$password" | curl -k -o $logfile2 -K -

{
echo
date
echo "intial curl has run" 
echo "updated here, pausing before enter loop"
} 2>&1   |  tee -a $logfile
sleep $waittime

# need to delay a little otherwise sometimes after the sleep at the end of the loop we're not checking Ip everytime
waittime=$((waittime - 5))

while :
do
now=$(date +%s) #current time in seconds
echo "Current value of now is" "$now" "Current value of runagain is" $runagain "if now is greater we should enter control loop" 2>&1   |  tee -a $logfile
    if  [[ "$now" > "$runagain" ]]
    then
    echo "now greater than runagain time" 2>&1   |  tee -a $logfile
    ipadd=$(curl ifconfig.co)
    ipad=$ipadd
        ipadd="${ipadd//./}"
        {
        echo "Updated IP is $ipad" 
        echo "Compare IP is $ipad2"
        } 2>&1   |  tee -a $logfile
          if [[ "$ipadd" -ne "$ipadd2" ]]
          then
          echo "IP addresses do not match update needed" 2>&1   |  tee -a $logfile
          echo url="https://api.dynu.com/nic/update?username=$username&password=$password" | curl -k -o $logfile2 -K -
            date 2>&1   |  tee -a $logfile2 
          date 2>&1   |  tee -a $logfile
          echo "updated here IP addresses" 2>&1   |  tee -a $logfile
          ipadd2=$ipadd
          ipad2=$ipadd2
          ((updates++))
          else
          echo "IP addresses match no update is needed" 2>&1   |  tee -a $logfile
          fi
          echo "updated variable and loop" 2>&1   |  tee -a $logfile
    fi
now=$(date +%s) #current time in seconds
runagain=$((now + waittime))
currenttime=$(date +%H:%M)
          if [[ $endtime < $currenttime ]] 
          then
{
echo
echo "Exiting reached" $endtime
echo "Final variables are"
echo "ipad is $ipad"
echo "ipadd is $ipadd"
echo "ipad2=$ipad2"
echo "ipadd2=$ipadd2"
echo
} 2>&1   |  tee -a $logfile
          break
          fi
          sleep $((waittime +5))
          {
          date
          echo "loops completed =" $counter
          echo "Latest IP is $ipad"
          echo "Compare IP is $ipad2"
          echo "IP has been updated " $updates " times."
          echo
          } 2>&1   |  tee -a $logfile
((counter++))
done
exit

Here's a complete logfile

+ date
Wed May 27 08:32:05 +08 2020
+ echo

+ echo 'next running intial curl, check logfile2'
next running intial curl, check logfile2
+ echo

+ echo

+ date
Wed May 27 08:32:07 +08 2020
+ echo 'intial curl has run'
intial curl has run
+ echo 'updated here, pausing before enter loop'
updated here, pausing before enter loop
Current value of now is 1590541327 Current value of runagain is 1590541323 if now is greater we should enter control loop
now greater than runagain time
+ echo 'Updated IP is 201.192.96.188'
Updated IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
IP addresses match no update is needed
updated variable and loop
+ date
Wed May 27 09:32:09 +08 2020
+ echo 'loops completed =' 0
loops completed = 0
+ echo 'Latest IP is 201.192.96.188'
Latest IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
+ echo 'IP has been updated ' 0 ' times.'
IP has been updated  0  times.
+ echo

Current value of now is 1590543129 Current value of runagain is 1590543124 if now is greater we should enter control loop
now greater than runagain time
+ echo 'Updated IP is 201.192.96.188'
Updated IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
IP addresses match no update is needed
updated variable and loop
+ date
Wed May 27 10:02:11 +08 2020
+ echo 'loops completed =' 1
loops completed = 1
+ echo 'Latest IP is 201.192.96.188'
Latest IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
+ echo 'IP has been updated ' 0 ' times.'
IP has been updated  0  times.
+ echo

Current value of now is 1590544931 Current value of runagain is 1590544926 if now is greater we should enter control loop
now greater than runagain time
+ echo 'Updated IP is 201.192.96.188'
Updated IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
IP addresses match no update is needed
updated variable and loop
+ date
Wed May 27 10:32:13 +08 2020
+ echo 'loops completed =' 2
loops completed = 2
+ echo 'Latest IP is 201.192.96.188'
Latest IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
+ echo 'IP has been updated ' 0 ' times.'
IP has been updated  0  times.
+ echo

Current value of now is 1590546733 Current value of runagain is 1590546728 if now is greater we should enter control loop
now greater than runagain time
+ echo 'Updated IP is 201.192.96.188'
Updated IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
IP addresses match no update is needed
updated variable and loop
+ date
Wed May 27 11:02:14 +08 2020
+ echo 'loops completed =' 3
loops completed = 3
+ echo 'Latest IP is 201.192.96.188'
Latest IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
+ echo 'IP has been updated ' 0 ' times.'
IP has been updated  0  times.
+ echo

Current value of now is 1590548534 Current value of runagain is 1590548529 if now is greater we should enter control loop
now greater than runagain time
+ echo 'Updated IP is 201.192.96.188'
Updated IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
IP addresses match no update is needed
updated variable and loop
+ date
Wed May 27 11:32:16 +08 2020
+ echo 'loops completed =' 4
loops completed = 4
+ echo 'Latest IP is 201.192.96.188'
Latest IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
+ echo 'IP has been updated ' 0 ' times.'
IP has been updated  0  times.
+ echo

Current value of now is 1590550336 Current value of runagain is 1590550331 if now is greater we should enter control loop
now greater than runagain time
+ echo 'Updated IP is 201.192.96.188'
Updated IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
IP addresses match no update is needed
updated variable and loop
+ date
Wed May 27 12:02:19 +08 2020
+ echo 'loops completed =' 5
loops completed = 5
+ echo 'Latest IP is 201.192.96.188'
Latest IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
+ echo 'IP has been updated ' 0 ' times.'
IP has been updated  0  times.
+ echo

Current value of now is 1590552139 Current value of runagain is 1590552134 if now is greater we should enter control loop
now greater than runagain time
+ echo 'Updated IP is 201.192.96.188'
Updated IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
IP addresses match no update is needed
updated variable and loop
+ date
Wed May 27 12:32:21 +08 2020
+ echo 'loops completed =' 6
loops completed = 6
+ echo 'Latest IP is 201.192.96.188'
Latest IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
+ echo 'IP has been updated ' 0 ' times.'
IP has been updated  0  times.
+ echo

Current value of now is 1590553941 Current value of runagain is 1590553936 if now is greater we should enter control loop
now greater than runagain time
+ echo 'Updated IP is 201.192.96.188'
Updated IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
IP addresses match no update is needed
updated variable and loop
+ date
Wed May 27 13:02:23 +08 2020
+ echo 'loops completed =' 7
loops completed = 7
+ echo 'Latest IP is 201.192.96.188'
Latest IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
+ echo 'IP has been updated ' 0 ' times.'
IP has been updated  0  times.
+ echo

Current value of now is 1590555743 Current value of runagain is 1590555738 if now is greater we should enter control loop
now greater than runagain time
+ echo 'Updated IP is <!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta charset="utf-8">
<title>Application Error</title>
<style media="screen">
          html,body,iframe {
            margin: 0;
            padding: 0;
          }
          html,body {
            height: 100%;
            overflow: hidden;
          }
          iframe {
            width: 100%;
            height: 100%;
            border: 0;
          }
        </style>
<script async src='\''/cdn-cgi/bm/cv/1284585713/api.js'\''></script></head>
<body>
<iframe src="//www.herokucdn.com/error-pages/application-error.html"></iframe>
<script type="text/javascript">(function(){window['\''__CF$cv$params'\'']={r:'\''599d2ef79d230000'\'',m:'\''84f41f65048b9d761b2b03ee2dba081e6fc5bc4b-1590555744-1800-ASehXqTGhFflf2FlurpjHewkGmp+xKuyFS6UqmwG34OrGiv4KMhbB9mjRD87nIr3FuSERG+Rx/OoGxPEAGHY48Cjc7iUDVBCDzugeAj7gV818FW7z10oWbSu/hYdlF8ILQ=='\'',s:[0xdc18430849,0xe865472e4f],}})();</script></body>
</html>'
Updated IP is <!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta charset="utf-8">
<title>Application Error</title>
<style media="screen">
          html,body,iframe {
            margin: 0;
            padding: 0;
          }
          html,body {
            height: 100%;
            overflow: hidden;
          }
          iframe {
            width: 100%;
            height: 100%;
            border: 0;
          }
        </style>
<script async src='/cdn-cgi/bm/cv/1284585713/api.js'></script></head>
<body>
<iframe src="//www.herokucdn.com/error-pages/application-error.html"></iframe>
<script type="text/javascript">(function(){window['__CF$cv$params']={r:'599d2ef79d230000',m:'84f41f65048b9d761b2b03ee2dba081e6fc5bc4b-1590555744-1800-ASehXqTGhFflf2FlurpjHewkGmp+xKuyFS6UqmwG34OrGiv4KMhbB9mjRD87nIr3FuSERG+Rx/OoGxPEAGHY48Cjc7iUDVBCDzugeAj7gV818FW7z10oWbSu/hYdlF8ILQ==',s:[0xdc18430849,0xe865472e4f],}})();</script></body>
</html>
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
IP addresses match no update is needed
updated variable and loop
+ date
Wed May 27 13:32:24 +08 2020
+ echo 'loops completed =' 8
loops completed = 8
+ echo 'Latest IP is <!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta charset="utf-8">
<title>Application Error</title>
<style media="screen">
          html,body,iframe {
            margin: 0;
            padding: 0;
          }
          html,body {
            height: 100%;
            overflow: hidden;
          }
          iframe {
            width: 100%;
            height: 100%;
            border: 0;
          }
        </style>
<script async src='\''/cdn-cgi/bm/cv/1284585713/api.js'\''></script></head>
<body>
<iframe src="//www.herokucdn.com/error-pages/application-error.html"></iframe>
<script type="text/javascript">(function(){window['\''__CF$cv$params'\'']={r:'\''599d2ef79d230000'\'',m:'\''84f41f65048b9d761b2b03ee2dba081e6fc5bc4b-1590555744-1800-ASehXqTGhFflf2FlurpjHewkGmp+xKuyFS6UqmwG34OrGiv4KMhbB9mjRD87nIr3FuSERG+Rx/OoGxPEAGHY48Cjc7iUDVBCDzugeAj7gV818FW7z10oWbSu/hYdlF8ILQ=='\'',s:[0xdc18430849,0xe865472e4f],}})();</script></body>
</html>'
Latest IP is <!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta charset="utf-8">
<title>Application Error</title>
<style media="screen">
          html,body,iframe {
            margin: 0;
            padding: 0;
          }
          html,body {
            height: 100%;
            overflow: hidden;
          }
          iframe {
            width: 100%;
            height: 100%;
            border: 0;
          }
        </style>
<script async src='/cdn-cgi/bm/cv/1284585713/api.js'></script></head>
<body>
<iframe src="//www.herokucdn.com/error-pages/application-error.html"></iframe>
<script type="text/javascript">(function(){window['__CF$cv$params']={r:'599d2ef79d230000',m:'84f41f65048b9d761b2b03ee2dba081e6fc5bc4b-1590555744-1800-ASehXqTGhFflf2FlurpjHewkGmp+xKuyFS6UqmwG34OrGiv4KMhbB9mjRD87nIr3FuSERG+Rx/OoGxPEAGHY48Cjc7iUDVBCDzugeAj7gV818FW7z10oWbSu/hYdlF8ILQ==',s:[0xdc18430849,0xe865472e4f],}})();</script></body>
</html>
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
+ echo 'IP has been updated ' 0 ' times.'
IP has been updated  0  times.
+ echo

Current value of now is 1590557544 Current value of runagain is 1590557539 if now is greater we should enter control loop
now greater than runagain time
+ echo 'Updated IP is 201.192.96.188'
Updated IP is 201.192.96.188
+ echo 'Compare IP is 201.192.96.188'
Compare IP is 201.192.96.188
IP addresses match no update is needed
updated variable and loop

It is not a huge deal but any suggestions on what I've missed would be most helpful.

What you have missed is that since you rely on an external service (the one you use curl against), you must validate the response from it. It obviously sometimes fail and your script must be able to cope with this. Unfortunately there's no way of knowing what "application error" they had (if it's their way of saying "you query us too often"), but your script simply can't rely on it being fool-proof. This is not a real answer because your code has multiple other issues and I'm not showing how to solve the immediate issue of getting an error back from that IP address service. — Kusalananda
– Kusalananda ♦, Commented May 27, 2020 at 6:38
Thanks. Didn't see the wood for the trees as it were. I'll work on a way of validating the response. — boots
– boots, Commented May 27, 2020 at 22:30

boots · Accepted Answer · 2020-06-05 03:09:11Z

Thanks to Kusalananda for pointing out the error in my reasoning. I have a solution to my immediate issue. I'll post it as the answer because although inelegant it does solve my problem and might thereby help someone else.

In short I made a function that has a choice of 9 possible source for getting the external IP, in place of the individual statements in the first script I call the function. Ideally since it always uses the first choice unless I'd like to be using a different one each time but I cannot see a simple way of achieving that.

#!/bin/bash
# Going to be saved in the scripts directory as function_get_ip.sh then can be included in another script
# just by sourcing this one e.g after the variable definitions in the other script have the line
# source function_get_ip.sh

#define a function to get the current external IP
RetrieveIP1="curl wtfismyip.com/text"
RetrieveIP2="curl ifconfig.me"
RetrieveIP3="curl ifconfig.co"
RetrieveIP4="curl 'https://api.ipify.org?format=txt'"
RetrieveIP5="curl http://wooledge.org/myip.cgi"
RetrieveIP6="curl http://CheckIP=.dyndns.org  2> /dev/null| perl -pe  's,.*Address: (\d+\.\d+\.\d+\.\d+).*,$1,'"
RetrieveIP7="wget -qO - http://ipecho.net/plain | xargs echo"
RetrieveIP8="wget -qO - icanhazip.com"
RetrieveIP9="wget -qO - ipv4bot.whatismyipaddress.com"
CheckIP=0

function get_ip {
                                  IP=$($RetrieveIP1) 
                                  if [[ $IP =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
                                  then
                                  CheckIP=1
                                  else
                              IP=$($RetrieveIP2) 
                              if [[ $IP =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
                              then
                              CheckIP=2
                              else
                          IP=$($RetrieveIP3) 
                          if [[ $IP =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
                          then
                          CheckIP=3
                          else
                      IP=$($RetrieveIP4) 
                      if [[ $IP =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
                      then
                      CheckIP=4
                      else
                  IP=$($RetrieveIP5) 
                  if [[ $IP =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
                  then
                  CheckIP=5
                  else
              IP=$($RetrieveIP6) 
              if [[ $IP =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
              then
              CheckIP=6
              else
          IP=$($RetrieveIP7) 
          if [[ $IP =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
          then
          CheckIP=7
          else
      IP=$($RetrieveIP8) 
      if [[ $IP =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
      then
      CheckIP=8
      else
  IP=$($RetrieveIP9) 
  if [[ $IP =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]
  then
  CheckIP=9
                  fi
                fi
              fi
            fi
          fi  
        fi
      fi
    fi
  fi
}

Calling the function give an output of $CheckIP with a non zero value if successful and $IP with the address

Stack Exchange Network

Log from running script sometimes appears as if it is HTML

1 Answer 1

You must log in to answer this question.

Hot Network Questions

Log from running script sometimes appears as if it is HTML

1 Answer 1

You must log in to answer this question.

Related

Hot Network Questions