convert mongodb query (unicode) to json using json_util

Question

from bson.json_util import dumps
def json_response(response):
return {"response":dumps(response,ensure_ascii=False).encode("utf8")
        ,"headers":{"Content-type":"text/json"}}

This problem is making me crazy. It returns an error randomly, and I can't find the solution.

/core/handlers/wsgi.py", line 38, in __call__, 
output = lookup_view(req), 
File "auth/decorator.py", line 8, in wrap, 
return fn(req,*args,**kwargs),
File "auth/decorator.py", line 21, in wrap, 
return fn(req,*args,**kwargs),
File "contrib/admin/views.py", line 67, in submit_base_premission,
return json_response({"baseperm":baseperm,"Meta":{"gmsg":u"...","type":201}}),
File "render/render_response.py", line 85, in json_response, 
return {"response":dumps(response,ensure_ascii=False).encode("utf8"),
File "/usr/local/lib/python2.7/dist-packages/bson/json_util.py", line 116, in dumps,
return json.dumps(_json_convert(obj), *args, **kwargs), 
File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps, referer: 
**kw).encode(obj), 
File "/usr/lib/python2.7/json/encoder.py", line 201, in encode, 
chunks = self.iterencode(o, _one_shot=True), 
File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode, 
return _iterencode(o, 0), 
File "/usr/lib/python2.7/json/encoder.py", line 178, in default, 
raise TypeError(repr(o) + " is not JSON serializable"), 
TypeError: ObjectId('51f7dcee95113b7a48e974fe') is not JSON serializable,

baseperm is a pymongo Cursor, it returns this error randomly and that is where I have the problem. It seems that it doesn't detect objectid sometimes and doesn't convert it to str so json raises an error on dumps.

Ioannis Lalopoulos · Accepted Answer · 2013-08-02 14:24:06Z

1

Check the version of the pymongo driver, if it is under version 2.4.2+ then you may need to update it. Before that version the __str__ method of ObjectId was handled incorrectly for 2.x versions of python, check the repo: github, ObjectId.__str__ should return str in 2.x.

To check the pymongo driver version, type in the python shell:

import pymongo
print(pymongo.version)

UPDATE

I suppose you have tested both environments with the same dataset, so give a try to upgrade python 2.7.3 to 2.7.5.

Else try to iterate through the cursor and construct the list before giving it to json_response() i.e.:

baseperm = list(baseperm) #now baseperm is a list of the documents

...
my_response['baseperm'] = baseperm
my_response['Meta'] = ...
...

return json_response(my_response)

edited Aug 2, 2013 at 14:24

answered Aug 1, 2013 at 23:11

Ioannis Lalopoulos

1,5111 gold badge12 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

hosein Over a year ago

print(pymongo.version) return 2.5.2 and i use Python 2.7.3

hosein Over a year ago

i found have this problem just on server using apache python 2.7.3 'uname -a' 'Linux ... 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux', on local everything work well ,'uname -a' Linux ... 3.9.9-302.fc19.x86_64 #1 SMP Sat Jul 6 13:41:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux python 2.7.5

hosein · Accepted Answer · 2013-08-06 05:39:35Z

I report this problem on mongodb issue tracker https://jira.mongodb.org/browse/PYTHON-548

answer:

You said this only happens occasionally? The only thing I can think of that might be related is mod_wsgi spawning sub interpreters. In PyMongo that tends to cause problems with the C extensions encoding python dicts to BSON. In your case this seems to be happening after the BSON documents are decoded to python dicts. It looks like isinstance is failing to match ObjectId in json_util.default(). PYTHON-539 seemed to be a similar problem related to some package miss configuration in the user's environment.

There could be a fairly large performance hit, but could you try running PyMongo without C extensions to see if that solves the problem?

You can read about the mod_wsgi issue here:

http://api.mongodb.org/python/current/faq.html#does-pymongo-work-with-mod-wsgi

Collectives™ on Stack Overflow

convert mongodb query (unicode) to json using json_util

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related