10

I am having a shell script which opens a file and pass it to python script for processing it. So if there is any issue with the file ( e.g. the file content is not in the format required for the successful execution for python script), the python script throws an exception. Since my objective it to process N files using python script. I need to know which is the file causing the script to break. I read on how to catch the exception in thrown by command execution. http://linuxcommand.org/wss0150.php . But in my case its the python script which throws the exception and I need to know in shell script what the exception has been thrown. Can anyone help me how can I proceed with this?

Below is the code snippet :

#!/bin/bash
yesterday=$(date --date "yesterday" "+%y%m%d")
ftoday=$(date --date "today" "+%m-%d-%Y")
year=$(date "+%Y")
fileList=$(find C:/logdata/$year/$ftoday/ -iname "*"$yesterday".log")
for var in $fileList
do
echo -e "\n START Processing File : $var" >> shelltestlog.txt
cat $var| ./scriptA.py 
echo -e "\n END Processing File : $var" >> shelltestlog.txt
done
Improve this question
1
  • If you just want to stop the script when it finds exception and debug right away. Then you can use syntax: ./scriptA.py || exit 1 Commented Oct 22, 2018 at 15:21

5 Answers 5

Reset to default
5

If your python script returns an non-zero error level whenever it gets an exception, you can use || { } to log messages:

./scriptA.py < "$file" || {
    printf "\n Python script scriptA.py failed with file \"%s\".\n" "$file" >> shelltestlog.txt
}

I actually tried to simplify your code first:

#!/bin/bash

yesterday=$(date --date "yesterday" "+%y%m%d")
ftoday=$(date --date "today" "+%m-%d-%Y")
year=$(date "+%Y")

readarray -t filesList < <(find C:/logdata/$year/$ftoday/ -iname "*"$yesterday".log")

for file in "${filesList[@]}"; do
    printf "\n START Processing File : %s\n" "$file" >> shelltestlog.txt
    ./scriptA.py < "$file" || {
        printf "\n Python script scriptA.py failed with file \"%s\".\n" "$file" >> shelltestlog.txt
    }
    printf "\n END Processing File : %s\n" "$file" >> shelltestlog.txt
done
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Thanks. Using above approach , I solved the problem of getting information about the failing file/s. But I was not able to catch the exception thrown by .py script.
@Codelearner Exceptions don't propagate across process boundaries. This is the best you can do short of rewriting the entire thing as one Python script as I suggested.
@Zack, I agree with you. Maybe the approach suggested by Chepner is helpful to get to know the exception details...
2

An uncaught exception will produce a traceback printed to standard error. The best you can do is capture that, and try to parse it.

if ! ./scriptA.py < "$var" 2> stderr.txt; then
  # Parse stderr.txt to see what exception was raised.
fi
Improve this answer

Comments

1

You can play some redirection games:

# open file descriptor 3, pointing to the same destination as stdout
exec 3>&1

# run the script, 
# python's stderr goes to stdout so it can be captured in a variable
# python's stdout goes to file descriptor 3, displayed on the terminal
python_stderr=$( ./scriptA.py < "$var" 2>&1 1>&3 )

# close the file descriptor
exec 3>&-

Then you can examine $python_stderr for patterns

case $python_stderr in
    "") echo "nothing on stderr" ;;
    *foo*) handle_foo_error ;;
    *bar*) handle_bar_error ;;
    *baz*) handle_baz_error ;;
esac
Improve this answer

1 Comment

It's important to do the redirections in the left-to-right order shown.
1

You should rewrite the shell script in Python and merge it into the existing Python script. The stuff that you are currently doing with the date command, can be done with the datetime and time modules. The stuff you are doing with find can be done with os.walk and fnmatch.fnmatch.

Here's an outline:

#! /usr/bin/python

def scriptA(file_to_process):
    # the code currently in scriptA goes here;
    # modify it to read from `file_to_process` rather than `sys.stdin`

def get_file_list():
    # your bash logic to construct the file list goes here,
    # converted to Python as suggested above

def main():
   for filename in get_file_list():
       sys.stderr.write("Processing {}...\n".format(filename))
       try:
           scriptA(open(filename, "rt"))

       except SomeExceptionType as e:
           # handle exception...

       except SomeOtherExceptionType as e:
           # handle other kind of exception...

       sys.stderr.write("Done processing {}.\n".format(filename))

main()
Improve this answer

1 Comment

Thank you for suggesting one more approach.
0

You can make use of a special $? variable in bash which contains the exit code of the previous command that was executed. Since you are running a python script inside bash, once the call ends it will generate some exit code.

Now, we need to manually access the exit code which can be found in bash $? variable which stores the exit codes. see more here

It is worth noting that successful execution of a process gives exit code 0. So any other exit code means there was some exception involved in the execution. So you can make use of if else here like this:

./scriptA.py
if [ $? -gt 0 ] ;then
    # echo "exit code $?"
    # do something with it
else
    # do something else
fi

This link is also helpful in understanding exit codes

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.