Python Debug Log Tool to trace execution AFTER program has finished?

Question

I want to build a tool that I can invoke when rolling out new code for testing that will capture and evaluate all of the lines of execution and print them to a file so that it can be traced and debugged long after execution has finished. Think of this as a debugger that evaluates the return of every line of code and prints that line of code plus the return values of that line in another column, as it is also running the code. For example:

Timestamp           File-Line    Source Code                       Debug Watcher        
----------------    ---------    -------------------------------   -----------------
20130109-10:18AM    test.py-1    import mathy  
20130109-10:18AM    test.py-2    x = 1        # inline comments   return: x:1
20130109-10:18AM    test.py-3    ans = divide(x,2)                call: math_functions.py-55:divide(dividend:1, divisor:2) 
20130109-10:18AM    mathy.py-56     return divedend / divisor     divide() function return: 0.5
20130109-10:18AM    test.py-4    print "Got %d"%ans               call: print : "Got 0.5"
etc...

Do you see how you can follow execution as it moves through separate functions, files, etc? This has the potential to generate a ton of log data, so it could be clipped to some limit size. I also have to punt on large data structures, such as dicts, arrays, etc. and create some special cases to handle those kinds of things.

I have created something like this before using traceback, but it never worked exactly right. Just wondering if something more polished might already exists. If not, how would you do it? Thanks.

Community · Accepted Answer · 2017-05-23 12:06:24Z

I would suggest to look into two places: tracing decorators and sys.settrace.

tracing decorators

I was using this approach to debug production code. Please note it doesn't log every line of python code executed and doesn't evaluate every involved variable. But what was enough for my task - it logs every function call, passed arguments and returned values. It uses logging framework so you may configure where this information will go - screen, logfile, etc. Also you'll need to attach tracer to module, class or method for it to start tracing.

Trace decorator for debugging

Here is short usage example:

import logging
logging.basicConfig(level=logging.DEBUG)

from trace_decorator import trace, attach

class TClass(object):
    def method(self,arg=1):
        print "using TClass.method()"
        self.method2(3)

    def method2(self,arg=2):
        d=1

@trace
def main():
    t = TClass()
    t.method(2)

And program output

DEBUG:main:>>> main()
DEBUG:DEBUG:>>> TClass.method( self = <TClass object at 0x25b87d0>, arg = 2 )
DEBUG:DEBUG:>>> TClass.method2( self = <TClass object at 0x25b87d0>, arg = 3 )
DEBUG:DEBUG:<<< TClass.method2
DEBUG:DEBUG:<<< TClass.method
DEBUG:main:<<< main

For more information and usage examples check the link with recipe.

sys.settrace

The tool your described sounds very like python debugger. To name few python debuggers: PyDev Eclipse Plugin has nice one) pdb But instead of doing interactive debugging when user chose where in code to evaluate some variables, you want to capture program state on every line. Anyway - python debugger would be the first place to look for the way to implement such tool.

Python debuggers use python's sys.settrace to put their trace handler method. There are many discussions on stackoverflow site on how to use sys.settrace method. But most of them has precautions against using this method - easy to break your program, using it will make your program run very slow.

Here is an python tracing tool based on sys.settrace - you may try to play with it. Please note you'll need python 2.7 and program should be started from command line strictly not some IDE or interactive shell. https://code.google.com/p/pytracemonitor/

Collectives™ on Stack Overflow

Python Debug Log Tool to trace execution AFTER program has finished?

1 Answer 1

tracing decorators

sys.settrace

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

tracing decorators

sys.settrace

Comments

Your Answer

Sign up or log in

Post as a guest

Related