15

How can I perform bitwise queries on the DB with Django?
I haven't found anything about it in the docs.
Should I retrieve a queryset and then filter programmically?

If you're interested, I use bitwise ops as an alternative to IN() statements in very large and complex queries, in order to improve performance.
I have a DB containing millions of items (records). Some fields use binary representation of an item property.
For example: the Color field can have multiple values, so it is structured like so:

0001 - Red
0010 - Green
0100 - Blue
1000 - White

(these are binary values)
So if an item has Red and Blue colors, the Color field will contain 0101.
When a user queries the DB, I use bitwise-OR to find matches (instead of IN() which is very slow).

4 Answers 4

12

You can perform database-level bitwise operations with F objects.

If field is non-negative, it means that condition field & mask > 0 can be re-written as (field > 0) AND (field >= (field & mask)). If you want to check if all bits of mask apply ((field & mask) == mask), you can build previous expression for each bit and then merge conditions via sql AND. Please see example how it can be done. (Custom QuerySet is just for convenience. If you use old django versions you can implement has_one_of and has_all as separate functions or classmethods, or better PathThroughManager). Note 1 * F is a workaround to force parenthesis over bitwise operation, otherwise django (as for version 1.5) will produce bad sql (colors >= colors & mask, comparison has higher priority, so it will mean TRUE & mask)

import operator
from django.db import models
from django.db.models import Q, F

_bit = lambda x: 2**(x-1)
RED = _bit(1)
GREEN = _bit(2)
BLUE = _bit(3)
WHITE = _bit(4)


class ItemColorsQuerySet(models.QuerySet):

    def has_one_of(self, colors):
        """
            Only those that has at least one of provided colors
        """
        return self.filter(
            colors__gt=0,
            # field value contains one of supplied color bits
            colors__lt=F('colors') + (1 * F('colors').bitand(reduce(operator.or_, colors, 0)))
        )

    def has_all(self, colors):
        """
            Has all provided colors (and probably others)
        """
        # filter conditions for all supplied colors: 
        # each one is "field value has bit that represents color"
        colors_q = map(lambda c: Q(colors__gte=1 * F('colors').bitand(c)), colors)
        # one complex Q object merged via sql AND:
        # colors>0 and all color-bit conditions
        filter_q = reduce(operator.and_, colors_q, Q(colors__gt=0))
        return self.filter(filter_q)


class Item(models.Model):

    name = models.CharField(max_length=100, unique=True)
    # can handle many colors using bitwise logic. Zero means no color is set.
    colors = models.PositiveIntegerField(default=0)

    objects = ItemColorsQuerySet.as_manager()
Sign up to request clarification or add additional context in comments.

6 Comments

Yes, that's was my own idea, I found this question while searching for solution, and when I finally implemented it in our project, just came back to share the idea.
I think it does not work if F('colors').bitand(color) is 0. I used flags__lt=(F("flags")+1*F("flags").bitand(flag))) instead
@Elisha Looks like you miss parentheses around +1
@IvanKlass maybe, I don't remember now what I tried to do.Thanks anyway :)
I don't think this is correct. For example, imagine the value of the field of the object in the db is 1010. You filter for 1000 -> correct, since the value in the db is larger than the value of the bit-wise and. But if you filter for 0001, you will also have the item returned, since 1010 & 0001 = 0000 < 1010. Or am I wrong? In my case only @gaozhidf's answer using annotate worked!
|
11

same like @Ivan Klass,

You can use bitand or bitor from django orm F,

# filter Red and Blue(0101)
Color.objects.annotate(
  color_filter=F('color').bitand(0101)
).filter(color_filter__gte=0101)

 # filter Red or Blue(0101)
Color.objects.annotate(
  color_filter=F('color').bitand(0101)
).filter(color_filter__gt=0)   

2 Comments

Couldn't you just use .filter(color_filter=0101) in the first case for "Red and Blue"?
@MoRe, yes, you can just filter "Red and Blue"(0101) with .filter(color_filter=0101), and filter those includes "Red and Blue"(like 0101, 0111) with .filter(color_filter__gte=0101)
10

For postgres db you might use .extra() params with django orm.

For example:

SomeModel.objects.extra(where=['brand_label & 3 = 3'])

Comments

7

Check django-bitfield , it works well w/ PostgreSQL (probably also well MySQL)

2 Comments

Hi, I'm actually using MySQL (thinking of migrating to MongoDB, but it doesn't support bitwise queries ATM)
@user1102018 I've just checked the code, it should work on MySQL, because it uses normal integer field and normal bitwise & and | which are all supported by MySQL.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.