django custom function for filter query

Question

I've a field in my model called, test_data = models.TextField(...), and the model is called MyOrm

and this test_data contains data which is actually string, some holds JSON data, and some reference to blob-url.

Now I'm trying to streamline my data. So I want to filter all the MyOrm object whose test_data ain't JSON.

I'm just storing/trying to store some meta-data along with url, then I will convert them.

Can anyone suggest me a way to do so??

pseudocode:

select all my-orm where is_not_json(my-orm.test-data)

If you're open to a schema change to do this, maybe consider using a JSON field type instead of TextField? — sytech
– sytech, Commented Mar 10, 2022 at 20:42

willeM_ Van Onsem · Accepted Answer · 2022-03-11 07:00:35Z

2

There is no database function for this that I'm aware of. You can define your own, but then this is more database programming.

I think you have no other option than to enumerate over the MyOrm model objects, and check if you can JSON decode these, with:

import json

for item in MyOrm.objects.all():
    try:
        json.loads(item.test_data)
    except ValueError:
        # is invalid JSON, process
        # …
        pass

or if memory might be a problem, you can work with .iterator(…) ^[Django-doc]:

import json

for item in MyOrm.objects.all().iterator():
    try:
        json.loads(item.test_data)
    except ValueError:
        # is invalid JSON, process
        # …
        pass

edited Mar 11, 2022 at 7:00

answered Mar 10, 2022 at 20:31

willeM_ Van Onsem

482k33 gold badges483 silver badges624 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Danial Over a year ago

object.all() maybe takes a ton of memory... the server i'm running it, it's a 4gb server... can you suggest some efficient way? And also provide some link to that database programming you mentioned?

willeM_ Van Onsem Over a year ago

@Danial: see for example stackoverflow.com/questions/30187554/…

willeM_ Van Onsem Over a year ago

@Danial: if memory is an issue, you can work with .iterator() to process the items in batches.

Collectives™ on Stack Overflow

django custom function for filter query

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related