84

I have read in a number of threads that Python pickle/cPickle cannot pickle lambda functions. However the following code works, using Python 2.7.6:

import cPickle as pickle

if __name__ == "__main__":
    s = pickle.dumps(lambda x, y: x+y)
    f = pickle.loads(s)
    assert f(3,4) == 7

So what is going on? Or, rather, what is the limit of pickling lambdas?

[EDIT] I think i know why this code runs. I forgot (sorry!) i am running stackless python, which has a form of micro-threads called tasklets executing a function. These tasklets can be halted, pickled, unpickled and continued, so i guess (asked on the stackless mailing list) that it also provides a way to pickle function bodies.

Improve this question
5
  • 4
    Could not replicate - in 2.7.6 (on OS X 10.9.4) I get TypeError: can't pickle function objects. Commented Aug 17, 2014 at 10:59
  • try pickle.loads from an other script; I think you pickle the reference to lambda wich, in the same scope, is preserved in memory and called. Commented Aug 17, 2014 at 11:33
  • Just to know, which version of cPickle are you using? Commented Aug 17, 2014 at 11:37
  • @Lars: stackless does exactly what dill does, in general… the major difference is that stackless replaces the call stack in C, while dill tries to register serialization functions using ctypes to work at the C layer as possible. Stackless can serialize all objects. Commented Aug 18, 2014 at 20:38
  • cloudpickle is the way: github.com/cloudpipe/cloudpickle Commented Nov 28, 2022 at 18:49

6 Answers 6

Reset to default
95

Yes, python can pickle lambda functions… but only if you have something that uses copy_reg to register how to pickle lambda functions -- the package dill loads the copy_reg you need into the pickle registry for you, when you import dill.

Python 2.7.8 (default, Jul 13 2014, 02:29:54) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import dill  # the code below will fail without this line
>>> 
>>> import pickle
>>> s = pickle.dumps(lambda x, y: x+y)
>>> f = pickle.loads(s)
>>> assert f(3,4) == 7
>>> f
<function <lambda> at 0x10aebdaa0>

get dill here: https://github.com/uqfoundation

Improve this answer
Sign up to request clarification or add additional context in comments.

10 Comments

I've tried on python3 In [1]: import dill In [2]: import pickle In [3]: pickle.dumps(lambda x: (x+1, x+2)) --------------------------------------------------------------------------- PicklingError Traceback (most recent call last) <ipython-input-3-924e2f4cc7e0> in <module>() ----> 1 pickle.dumps(lambda x: (x+1, x+2)) PicklingError: Can't pickle <function <lambda> at 0x7f08ee40ca60>: attribute lookup <lambda> on main failed. It only works if you import dill as pickle
@Ramast: you are correct -- in python3, you currently have to import dill as pickle. In python2, what I have above works either way you do it.
@CharlieParker: Can you elaborate? The above should work for most "arbitrary" functions.
The answer is incorrect for Python 3.6 as it is - use dill.dumps() and dill.loads() instead.
@CiprianTomoiagă: The primary difference is that in python 2, there was pickle (python) and cPickle (C) -- and in python 3, it's now pickle (python) and _pickle (C)... however, pickle.dump (and dumps) uses _pickle (i.e. C)... and dill can currently only inject new methods into the python pickle registry. So, just importing dill doesn't work like it does in python 2. Note that there is pickle._dump (and _dumps), which does use the python registry... so that works as in python 2. Unfortunately most packages don't fall back to _dump, when dump fails.
|
49

Python can pickle lambdas. We will cover Python 2 and 3 separately as implementation of pickle are different in different Python versions.

  • Python 3.6

In Python 3, there is no module named cPickle. We have pickle instead which also doesn't support pickling of lambda functions by default. Let's see it's dispatch table:

>> import pickle
>> pickle.Pickler.dispatch_table
<member 'dispatch_table' of '_pickle.Pickler' objects>

Wait. I tried looking up dispatch_table of pickle not _pickle. _pickle is the alternative and faster C implementation of pickle. But we haven't imported it yet! This C implementation is imported automatically, if it is available, at the end of pure Python pickle module.

# Use the faster _pickle if possible
try:
    from _pickle import (
        PickleError,
        PicklingError,
        UnpicklingError,
        Pickler,
        Unpickler,
        dump,
        dumps,
        load,
        loads
    )
except ImportError:
    Pickler, Unpickler = _Pickler, _Unpickler
    dump, dumps, load, loads = _dump, _dumps, _load, _loads

We are still left with the question of pickling lambdas in Python 3. The answer is you CAN'T with the native pickle or _pickle. You will need to import dill or cloudpickle and use that instead of the native pickle module.

>> import dill
>> dill.loads(dill.dumps(lambda x:x))
<function __main__.<lambda>>
  • Python 2.7

pickle uses pickle registry which is nothing but a mapping from type to the function to use for serializing (pickling) objects of that type. You can see pickle registry as:

>> pickle.Pickler.dispatch

{bool: <function pickle.save_bool>,
 instance: <function pickle.save_inst>,
 classobj: <function pickle.save_global>,
 float: <function pickle.save_float>,
 function: <function pickle.save_global>,
 int: <function pickle.save_int>,
 list: <function pickle.save_list>,
 long: <function pickle.save_long>,
 dict: <function pickle.save_dict>,
 builtin_function_or_method: <function pickle.save_global>,
 NoneType: <function pickle.save_none>,
 str: <function pickle.save_string>,
 tuple: <function pickle.save_tuple>,
 type: <function pickle.save_global>,
 unicode: <function pickle.save_unicode>}

To pickle custom types, Python provides copy_reg module to register our functions. You can read more about it here. By default, copy_regmodule supports pickling of the following additional types:

>> import copy_reg
>> copy_reg.dispatch_table

{code: <function ipykernel.codeutil.reduce_code>,
 complex: <function copy_reg.pickle_complex>,
 _sre.SRE_Pattern: <function re._pickle>,
 posix.statvfs_result: <function os._pickle_statvfs_result>,
 posix.stat_result: <function os._pickle_stat_result>}

Now, type of lambda functions is types.FunctionType. However, the builtin function for this type function: <function pickle.save_global> is not able to serialize lambda functions. Therefore, all third party libraries like dill, cloudpickle, etc override the inbuilt method to serialize lambda functions with some additional logic. Let's import dill and see what it does.

>> import dill
>> pickle.Pickler.dispatch

{_pyio.BufferedReader: <function dill.dill.save_file>,
 _pyio.TextIOWrapper: <function dill.dill.save_file>,
 _pyio.BufferedWriter: <function dill.dill.save_file>,
 _pyio.BufferedRandom: <function dill.dill.save_file>,
 functools.partial: <function dill.dill.save_functor>,
 operator.attrgetter: <function dill.dill.save_attrgetter>,
 operator.itemgetter: <function dill.dill.save_itemgetter>,
 cStringIO.StringI: <function dill.dill.save_stringi>,
 cStringIO.StringO: <function dill.dill.save_stringo>,
 bool: <function pickle.save_bool>,
 cell: <function dill.dill.save_cell>,
 instancemethod: <function dill.dill.save_instancemethod0>,
 instance: <function pickle.save_inst>,
 classobj: <function dill.dill.save_classobj>,
 code: <function dill.dill.save_code>,
 property: <function dill.dill.save_property>,
 method-wrapper: <function dill.dill.save_instancemethod>,
 dictproxy: <function dill.dill.save_dictproxy>,
 wrapper_descriptor: <function dill.dill.save_wrapper_descriptor>,
 getset_descriptor: <function dill.dill.save_wrapper_descriptor>,
 member_descriptor: <function dill.dill.save_wrapper_descriptor>,
 method_descriptor: <function dill.dill.save_wrapper_descriptor>,
 file: <function dill.dill.save_file>,
 float: <function pickle.save_float>,
 staticmethod: <function dill.dill.save_classmethod>,
 classmethod: <function dill.dill.save_classmethod>,
 function: <function dill.dill.save_function>,
 int: <function pickle.save_int>,
 list: <function pickle.save_list>,
 long: <function pickle.save_long>,
 dict: <function dill.dill.save_module_dict>,
 builtin_function_or_method: <function dill.dill.save_builtin_method>,
 module: <function dill.dill.save_module>,
 NotImplementedType: <function dill.dill.save_singleton>,
 NoneType: <function pickle.save_none>,
 xrange: <function dill.dill.save_singleton>,
 slice: <function dill.dill.save_slice>,
 ellipsis: <function dill.dill.save_singleton>,
 str: <function pickle.save_string>,
 tuple: <function pickle.save_tuple>,
 super: <function dill.dill.save_functor>,
 type: <function dill.dill.save_type>,
 weakcallableproxy: <function dill.dill.save_weakproxy>,
 weakproxy: <function dill.dill.save_weakproxy>,
 weakref: <function dill.dill.save_weakref>,
 unicode: <function pickle.save_unicode>,
 thread.lock: <function dill.dill.save_lock>}

Now, let's try to pickle lambda function.

>> pickle.loads(pickle.dumps(lambda x:x))
<function __main__.<lambda>>

It WORKS!!

In Python 2 we have two versions of pickle -

import pickle # pure Python version
pickle.__file__ # <install directory>/python-2.7/lib64/python2.7/pickle.py

import cPickle # C extension
cPickle.__file__ # <install directory>/python-2.7/lib64/python2.7/lib-dynload/cPickle.so

Now, let's try to pickle lambda with C implementation cPickle.

>> import cPickle
>> cPickle.loads(cPickle.dumps(lambda x:x))
TypeError: can't pickle function objects

What went wrong? Let's see the dispatch table of cPickle.

>> cPickle.Pickler.dispatch_table
AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch_table'

The implementation of pickle and cPickle is different. Importing dill makes only Python version of pickle work. The disadvantage of using pickle instead of cPickle is that it can be as much as 1000 times slower than cPickle.

I hope this clears all the doubts.

Improve this answer

1 Comment

This answer should be accepted. It nicely explains the usability and limitations with the dill package in each python version. Well done!
35

No, Python can't pickle lambda functions:

>>> import cPickle as pickle
>>> s = pickle.dumps(lambda x,y: x+y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

Not sure what you did that succeeded...

Improve this answer

2 Comments

I don't know why this comment is down voted. pickle can't serialize lambdas only dill package can
Why can't python pickle lambdas?
2

Even though it might be obvious I would like to add an other possible solution. As you probably know lambda functions are just anonymous function declarations. If you don't have many lambdas that are used only once and it wouldn't add much noise to your code you could just name your lambda and pass the name of it (without the parentheses) like this:

import cPickle as pickle

def addition(x, y):
    return x+y


if __name__ == "__main__":
    s = pickle.dumps(addition)
    f = pickle.loads(s)
    assert f(3,4) == 7

The name also adds more semantic and you wouldn't need an additional dependency like Dill. But only do that if that outweighs the added noise of the additional function(s).

Improve this answer

Comments

2

Install dill

$ pip install dill

Touch a file

touch yeah.p

Now run this python3 script, 

import dill

dill.dump(lambda x:x+1, open('yeah.p', 'wb'))
my_lambda = dill.load(open('yeah.p', 'rb'))
print(my_lambda(2))  # 3
Improve this answer

Comments

2

what worked for me (windows 10, python 3.7) was to pass a function instead of a lambda function:

def merge(x):
    return Image.merge("RGB", x.split()[::-1])

transforms.Lambda(merge)

instead of:

transforms.Lambda(lambda x: Image.merge("RGB", x.split()[::-1]))

no dill or cPickel needed.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.