Redis Python - how to delete all keys according to a specific pattern In python, without python iterating

Question

I'm writing a django management command to handle some of our redis caching. Basically, I need to choose all keys, that confirm to a certain pattern (for example: "prefix:*") and delete them.

I know I can use the cli to do that:

redis-cli KEYS "prefix:*" | xargs redis-cli DEL

But I need to do this from within the app. So I need to use the python binding (I'm using py-redis). I have tried feeding a list into delete, but it fails:

from common.redis_client import get_redis_client
cache = get_redis_client()
x = cache.keys('prefix:*') 

x == ['prefix:key1','prefix:key2'] # True

# And now

cache.delete(x)

# returns 0 . nothing is deleted

I know I can iterate over x:

for key in x:
   cache.delete(key)

But that would be losing redis awesome speed and misusing its capabilities. Is there a pythonic solution with py-redis, without iteration and/or the cli?

Thanks!

Alex Toderita · Accepted Answer · 2016-05-10 09:59:28Z

66

Use SCAN iterators: https://pypi.python.org/pypi/redis

for key in r.scan_iter("prefix:*"):
    r.delete(key)

answered May 10, 2016 at 9:59

Alex Toderita

6611 gold badge5 silver badges2 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Robert Lujo Over a year ago

django-redis implements delete_pattern which does something very similar to this, see github.com/niwinz/django-redis/blob/master/django_redis/client/… .

Dan Nissenbaum Over a year ago

I cannot find scan_iter in the Redis Python documentation. Has it been removed?

radtek · Accepted Answer · 2020-01-08 17:17:00Z

47

Here is a full working example using py-redis:

from redis import StrictRedis
cache = StrictRedis()

def clear_ns(ns):
    """
    Clears a namespace
    :param ns: str, namespace i.e your:prefix
    :return: int, cleared keys
    """
    count = 0
    ns_keys = ns + '*'
    for key in cache.scan_iter(ns_keys):
        cache.delete(key)
        count += 1
    return count

You can also do scan_iter to get all the keys into memory, and then pass all the keys to delete for a bulk delete but may take a good chunk of memory for larger namespaces. So probably best to run a delete for each key.

Cheers!

UPDATE:

Since writing the answer, I started using pipelining feature of redis to send all commands in one request and avoid network latency:

from redis import StrictRedis
cache = StrictRedis()

def clear_cache_ns(ns):
    """
    Clears a namespace in redis cache.
    This may be very time consuming.
    :param ns: str, namespace i.e your:prefix*
    :return: int, num cleared keys
    """
    count = 0
    pipe = cache.pipeline()
    for key in cache.scan_iter(ns):
        pipe.delete(key)
        count += 1
    pipe.execute()
    return count

UPDATE2 (Best Performing):

If you use scan instead of scan_iter, you can control the chunk size and iterate through the cursor using your own logic. This also seems to be a lot faster, especially when dealing with many keys. If you add pipelining to this you will get a bit of a performance boost, 10-25% depending on chunk size, at the cost of memory usage since you will not send the execute command to Redis until everything is generated. So I stuck with scan:

from redis import StrictRedis
cache = StrictRedis()
CHUNK_SIZE = 5000

def clear_ns(ns):
    """
    Clears a namespace
    :param ns: str, namespace i.e your:prefix
    :return: int, cleared keys
    """
    cursor = '0'
    ns_keys = ns + '*'
    while cursor != 0:
        cursor, keys = cache.scan(cursor=cursor, match=ns_keys, count=CHUNK_SIZE)
        if keys:
            cache.delete(*keys)

    return True

Here are some benchmarks:

5k chunks using a busy Redis cluster:

Done removing using scan in 4.49929285049
Done removing using scan_iter in 98.4856731892
Done removing using scan_iter & pipe in 66.8833789825
Done removing using scan & pipe in 3.20298910141

5k chunks and a small idle dev redis (localhost):

Done removing using scan in 1.26654982567
Done removing using scan_iter in 13.5976779461
Done removing using scan_iter & pipe in 4.66061878204
Done removing using scan & pipe in 1.13942599297

edited Jan 8, 2020 at 17:17

answered Jul 17, 2017 at 20:55

radtek

36.6k13 gold badges149 silver badges114 bronze badges

9 Comments

radtek Over a year ago

I provided a full working example. Also would like others to comment on scan_iter vs bulk delete

Blackeagle52 Over a year ago

Great answer, this should be the correct answer. Today I actually needed this answer myself and am preferring your answer above mine. Although some minor errors in your example, like no ns_keys variable in your first update, and :: within your second update.

radtek Over a year ago

Thanks, but I actually don't use scanning in prod because its so slow, instead I end up caching every key in the namespace and doing a bulk delete that way. Seems like overkill I know but the performance is best because you don't have to scan cache at all.

Jivan Over a year ago

The typing between cursor = '0' and while cursor != 0 is awkward. You could use a cursor = None and cache.scan(cursor=cursor or 0, ...) to make it slightly better

Kyle Barron Over a year ago

Note that even when this answer was written in 2017, redis-py has allowed you to provide a chunksize to scan_iter. You don't have to manage the cursor yourself. (github.com/andymccurdy/redis-py/blob/…)

|

Community · Accepted Answer · 2017-05-23 12:26:31Z

28

I think the

 for key in x: cache.delete(key)

is pretty good and concise. delete really wants one key at a time, so you have to loop.

Otherwise, this previous question and answer points you to a lua-based solution.

edited May 23, 2017 at 12:26

CommunityBot

11 silver badge

answered Feb 23, 2014 at 22:17

Dirk is no longer here

370k60 gold badges668 silver badges742 bronze badges

1 Comment

selfnamed Over a year ago

Using redis-python package you can do that such way: cache.delete(*keys)

James · Accepted Answer · 2016-05-17 02:34:40Z

12

From the Documentation

delete(*names)
    Delete one or more keys specified by names

This just wants an argument per key to delete and then it will tell you how many of them were found and deleted.

In the case of your code above I believe you can just do:

    redis.delete(*x)

But I will admit I am new to python and I just do:

    deleted_count = redis.delete('key1', 'key2')

answered May 17, 2016 at 2:34

James

1,6632 gold badges19 silver badges24 bronze badges

Comments

Gleb · Accepted Answer · 2016-02-25 14:15:54Z

10

Btw, for the django-redis you can use the following (from https://niwinz.github.io/django-redis/latest/):

from django.core.cache import cache
cache.delete_pattern("foo_*")

answered Feb 25, 2016 at 14:15

Gleb

7611 gold badge8 silver badges15 bronze badges

1 Comment

radtek Over a year ago

This question pertains to redis, so django cache framework is out of scope.

Nemo · Accepted Answer · 2018-12-20 22:15:20Z

8

You can use a specific pattern to match all keys and delete them:

import redis
client = redis.Redis(host='192.168.1.106', port=6379,
                password='pass', decode_responses=True)
for key in client.keys('prefix:*'):
    client.delete(key)

edited Dec 20, 2018 at 22:15

Nemo

2,6022 gold badges32 silver badges65 bronze badges

answered Sep 12, 2018 at 9:55

Lynn Han

5036 silver badges8 bronze badges

Comments

Nam G VU · Accepted Answer · 2017-12-04 13:57:35Z

7

cache.delete(*keys) solution of Dirk works fine, but make sure keys isn't empty to avoid a redis.exceptions.ResponseError: wrong number of arguments for 'del' command.

If you are sure that you will always get a result: cache.delete(*cache.keys('prefix:*') )

edited Dec 4, 2017 at 13:57

Nam G VU

35.8k79 gold badges248 silver badges398 bronze badges

answered Oct 31, 2014 at 15:03

Blackeagle52

1,98517 silver badges16 bronze badges

1 Comment

radtek Over a year ago

Don't use cache.keys() in prod, its intended for debugging: redis.io/commands/keys

Nemo · Accepted Answer · 2018-12-20 22:16:52Z

4

According to my test, it will costs too much time if I use scan_iter solution (as Alex Toderita wrote).

Therefore, I prefer to use:

from redis.connection import ResponseError

try:
    redis_obj.eval('''return redis.call('del', unpack(redis.call('keys', ARGV[1])))''', 0, 'prefix:*')
except ResponseError:
    pass

The prefix:* is the pattern.

refers to: https://stackoverflow.com/a/16974060

edited Dec 20, 2018 at 22:16

Nemo

2,6022 gold badges32 silver badges65 bronze badges

answered Nov 7, 2017 at 2:35

carton.swing

1,77719 silver badges12 bronze badges

Comments

Pankaj Saini · Accepted Answer · 2023-02-16 12:18:59Z

The answer suggested by @radtek is not working for me, since the keys are getting deleted while iterating, which leads to unexpected behavior. Here's an example:

from redis import StrictRedis
cache = StrictRedis()

for i in range(0, 10000):
    cache.set(f'test_{i}', 1)


cursor = '0'
SCAN_BATCH_SIZE = 5000
while cursor != 0:
    cursor, keys = self._redis.scan(cursor=cursor, match='test_*', count=SCAN_BATCH_SIZE)
    if keys:
        cache.delete(*keys)


## Iteration 1
# cursor=5000, keys=['test_0', .... , 'test_4999']
# keys will get deleted

## Iteration 2
# cursor=0, keys=[]
# No remaining keys are found reason being, there are just the 
# 5000 entries left post deletion and the cursor position is already 
# at 5000. Hence, no keys are returned.

You can use redis pipeline in order to solve this issue as mentioned below:

from redis import StrictRedis
cache = StrictRedis()

for i in range(0, 10000):
    cache.set(f'test_{i}', 1)


pipe = cache.pipeline()
cursor = None
SCAN_BATCH_SIZE = 5000
while cursor != 0:
    cursor, keys = self._redis.scan(cursor=cursor or 0, match='test_*', count=SCAN_BATCH_SIZE)
    if keys:
        pipe.delete(*keys)
pipe.execute()

Stephen Rauch · Accepted Answer · 2017-12-15 04:59:27Z

0

Use delete_pattern: https://niwinz.github.io/django-redis/latest/

from django.core.cache import cache
cache.delete_pattern("prefix:*")

edited Dec 15, 2017 at 4:59

Stephen Rauch♦

50.1k32 gold badges118 silver badges143 bronze badges

answered Dec 15, 2017 at 4:41

Jijo

1575 bronze badges

Comments

Fernando Valente · Accepted Answer · 2025-07-15 13:33:39Z

0

Use flushall() to clean all data from Redis cache

answered Jul 15 at 13:33

Fernando Valente

3272 silver badges9 bronze badges

Collectives™ on Stack Overflow

Redis Python - how to delete all keys according to a specific pattern In python, without python iterating

11 Answers 11

2 Comments

9 Comments

1 Comment

Comments

1 Comment

Comments

1 Comment

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

11 Answers 11

2 Comments

9 Comments

1 Comment

Comments

1 Comment

Comments

1 Comment

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related