Python weakref callbacks and del execution order

Question

In Python, is there a way to call a function after an object is finalized?

I thought the callback in a weakref would do it, but it appears a weakref's callback is called once the object is garbage collected, but before the objects __del__ method is called. This seems contrary to the notes on weakrefs and garbage collection in the Python trunk. Here's an example.

import sys
import weakref

class Spam(object) :
  def __init__(self, name) :
    self.name = name

  def __del__(self) :
    sys.stdout.write("Deleting Spam:%s\n" % self.name)
    sys.stdout.flush()

def cleaner(reference) :
  sys.stdout.write("In callback with reference %s\n" % reference)
  sys.stdout.flush()

spam = Spam("first")
wk_spam = weakref.ref(spam, cleaner)
del spam

The output I get is

$ python weakref_test.py 
In callback with reference <weakref at 0xc760a8; dead>
Deleting Spam:first

Is there some other conventional way to do what I want? Can I somehow force the finalization in my callback?

For those curious, I'm trying to wrap PyTables access so when all the datasets are no longer in use, the file automatically closes, and I avoid the "Closing remaining open files..." messages. (This is for library code abstracting the reading of hundreds of files.) — AFoglia
– AFoglia, Commented Aug 4, 2009 at 17:21

Jochen Ritzel · Accepted Answer · 2009-08-04 17:40:40Z

3

If "do what you want" means "run code when a resource leaves the context" ( instead of, for example, "abuse the garbage collector to do stuff" ), you're looking in the wrong direction. Python abtracted that whole idea as context-managers, used with the with statement.

from __future__ import with_statement
import sys
class Spam(object):
    def __init__(self, name):
        self.name = name

    def __enter__(self):
        sys.stdout.write("Entering Spam:%s\n" % self.name)
        sys.stdout.flush()

    def __exit__(self, type, value, traceback):
        sys.stdout.write("Lets clean up Spam:%s\n" % self.name) 
        if type is None:
            sys.stdout.write("Leaving Spam:%s in peace\n" % self.name)
            return
        else:
            sys.stdout.write("Leaving Spam:%s with Exception:%r\n" % (self.name, value))


with Spam("first") as spam:
    pass

with Spam("2nd") as spam:
    raise Exception("Oh No!")

gives:

Entering Spam:first
Lets clean up Spam:first
Leaving Spam:first in peace
Entering Spam:2nd
Lets clean up Spam:2nd
Leaving Spam:2nd with Exception:Exception('Oh No!',)
Traceback (most recent call last):
  File "asd.py", line 24, in <module>
    raise Exception("Oh No!")
Exception: Oh No!

answered Aug 4, 2009 at 17:40

Jochen Ritzel

108k33 gold badges205 silver badges196 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

AFoglia Over a year ago

with statements work fine when you only use a resource for a short amount of time, but for creating a resource which the client can use however and pass between multiple functions, it is very clumsy. Imagine if you could only get a file object via with statement. That wouldn't be pleasant.

AFoglia Over a year ago

Also, I have no choice on the __del__. That's in third party code used to read the files (PyTables). I'm wrapping those objects because they're necessary coordination between the objects needed to read our layout. Think of it as if PyTables read and wrote ar files, and my code makes sure the ar files are in valid deb package layout. Now replace "ar" with "hdf5", and "deb package" with "our data". (And our layout consists of multiple files, not just one.) So removing the __del__ is not a realistic option.

S.Lott Over a year ago

@AFoglia: "with statements work fine when you only use a resource for a short amount of time" Hardly true. You can put any volume of code you want in the with statement. Only accessing a file object with a with statements makes complete sense. Using a dataset is done in a with block that encompasses a lot of functionality, including passing the dataset from function to function.

Unknown · Accepted Answer · 2009-08-04 19:52:40Z

Here is a solution that uses a serialized garbage loop on another thread. Its probably the closest solution you'll get.

import sys
from threading import Thread
from Queue import Queue
import weakref

alive = weakref.WeakValueDictionary()
dump = Queue()

def garbageloop():
    while True:
        f = dump.get()
        f()

garbage_thread = Thread(target=garbageloop)
garbage_thread.daemon = True
garbage_thread.start()

class Spam(object) :
  def __init__(self, name) :
    self.name = name
    alive[id(self)] = self

  def __del__(self) :
    sys.stdout.write("Deleting Spam:%s\n" % self.name)
    sys.stdout.flush()
    dump.put(lambda:cleaner(id(self)))

def cleaner(address):
  if address in alive:
    sys.stdout.write("Object was still alive\n")
  else:
    sys.stdout.write("Object is dead\n")
  sys.stdout.flush()

spam = Spam("first")
del spam

heiner · Accepted Answer · 2020-04-16 14:54:30Z

0

For a "more than you wanted to know" documentation on weakrefs, __del__ and garbage collection, check out Modules/gc_weakref.txt in the cpython source code.

answered Apr 16, 2020 at 14:54

heiner

7388 silver badges12 bronze badges

Collectives™ on Stack Overflow

Python weakref callbacks and del execution order

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related