Serializing output to JSON - ValueError: Circular reference detected

Question

I'm trying to output results of my mysql query to JSON. I have problem with serializing datetime.datetime field, so I wrote small function to do that:

def date_handler(obj):
    if hasattr(obj, 'isoformat'):
        return obj.isoformat()
    else:
        return obj

and then in main code I'm just running:

products_json = []
for code in best_matching_codes:
    cur = db.cursor()
    query = "SELECT * FROM %s WHERE code LIKE '%s'" % (PRODUCTS_TABLE_NAME, product_code)
    cur.execute(query)
    columns = [desc[0] for desc in cur.description]
    rows = cur.fetchall()
    for row in rows:
        products_json.append(dict((k,v) for k,v in zip(columns,row)))   

return json.dumps(products_json, default = date_handler)

However, since I wrote date_handler function, I'm getting "ValueError: Circular reference detected"

127.0.0.1 - - [10/Jan/2013 00:42:18] "GET /1/product?code=9571%2F702 HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1701, in __call__
return self.wsgi_app(environ, start_response)
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1689, in wsgi_app
response = self.make_response(self.handle_exception(e))
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1687, in wsgi_app
response = self.full_dispatch_request()
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1360, in full_dispatch_request
rv = self.handle_user_exception(e)
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1358, in full_dispatch_request
rv = self.dispatch_request()
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1344, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/pisarzp/Desktop/SusyChoosy/susyAPI/test1.py", line 69, in product_search
return json.dumps(products_json, default = date_handler)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 238, in dumps
**kw).encode(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 201, in encode
chunks = self.iterencode(o, _one_shot=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 264, in iterencode
return _iterencode(o, 0)
ValueError: Circular reference detected

What did I break? Is there a better way to serialize output to JSON?

Could you get the print output for products_json? I guess there are some duplicated between the key with value on date. Then it can get the similar error. — jinghli
– jinghli, Commented Jan 10, 2013 at 2:23

Maxime Lorant · Accepted Answer · 2015-08-10 07:40:51Z

28

The function you pass as the default argument will only be called for objects that are not natively serializable by the json module. It must return a serializable object, or raise a TypeError.

Your version returns the same object you were passed if it's not of the one type you're fixing (dates). That is causing the circular reference error (which is misleading, since the circle is between one object and itself after being processed by date_handler).

You can start to fix this by changing date_handler to raise an exception in its else block. That will still probably fail, but you can probably then find out what object it is that is in your data structure causing the problem using code like this:

def date_handler(obj):
    if hasattr(obj, 'isoformat'):
        return obj.isoformat()
    else:
        raise TypeError(
            "Unserializable object {} of type {}".format(obj, type(obj))
        )

edited Aug 10, 2015 at 7:40

Maxime Lorant

36.4k19 gold badges91 silver badges97 bronze badges

answered Jan 10, 2013 at 5:18

Blckknght

106k11 gold badges135 silver badges188 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Azat Ibrakov · Accepted Answer · 2019-04-17 11:05:09Z

10

Instead of raising the TypeError yourself, you should relay the call to JSONEncoder's default-method:

def date_handler(obj):
    if hasattr(obj, 'isoformat'):
        return obj.isoformat()
    else:
        json.JSONEncoder.default(self,obj)

This will also raise TypeError and is a better practice, it allows for JSONEncoder to try and encode the type your method can't.

edited Apr 17, 2019 at 11:05

Azat Ibrakov

11.1k9 gold badges43 silver badges58 bronze badges

answered Jul 29, 2013 at 11:42

Joren Van Severen

2,3692 gold badges24 silver badges33 bronze badges

4 Comments

HereHere Over a year ago

I got NameError: global name 'self' is not defined

GBrian Over a year ago

Just try returning str(obj)

Hardik Gajjar Over a year ago

i have face same problem with array list inside json and change variable name then got error is gone

Jon Over a year ago

you would want to return your result in your else branch

GrvTyagi · Accepted Answer · 2019-06-10 07:17:15Z

2

json.dumps(obj, default=method_name)

"method_name" function must be return a serialize object.

def method_name(obj):
    data = {
            '__class__': obj.__class__.__name__,
            '__module__': obj.__module__
           }
    data.update(obj.__dict__)
    return data

edited Jun 10, 2019 at 7:17

answered Nov 2, 2015 at 17:52

GrvTyagi

4,5071 gold badge37 silver badges42 bronze badges

Comments

ThomazPom · Accepted Answer · 2023-03-23 14:02:59Z

0

Instead of raising an error you could simply Remove circular references in dicts, lists, tuples, making the object serializable

answered Mar 23, 2023 at 14:02

ThomazPom

1461 silver badge13 bronze badges

Collectives™ on Stack Overflow

Serializing output to JSON - ValueError: Circular reference detected

4 Answers 4

Comments

4 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

4 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related