I have a 2D array with 5000 Rows and 2 Million Columns the value stored is just a boolean 0 or 1. What is the Redis data structure best suited to storing this use-case in Redis?
Add a comment
|
2 Answers
A straightforward approach would be to store the array as a string and use bit commands to manipulate it. That should lead to minimal memory use, and SETBIT and GETBIT are both O(1).
If the data could fit in the 512MB limit of a Redis string you could just use a single string. The size you've given is too large, though, so an alternative approach would be to store each row as a separate string.
answered Jun 21, 2019 at 8:30
Kevin Christopher Henry
49.7k8 gold badges125 silver badges111 bronze badges
5 Comments
Leo Prince
sure , SETBIT seems to be interesting, could you please elaborate more on it? Thank you
Itamar Haber
That's a great answer but the only problem is that the entire array is 1.5GB and a bitmap can "only" be up to 0.5GB. The resolution can be to have each row of the array a bitmap.
Kevin Christopher Henry
@ItamarHaber: Thanks, for some reason I thought that limit had been lifted, but I guess not. I've updated the answer.
Itamar Haber
Not yet lifted, but "soon(tm)" ;P
Leo Prince
@KevinChristopherHenry Thank you!
You can make 5000 databases in a Redis instance, then save each row with the index of the columns which have the value of '1' instead of saving all of the 2 million columns. By doing this, maybe you can reduce the size of each row to half a million length. Then at the time of deserialization, you may build the initial value of array programmatically.
//example: row -0 - columns: 00101....1
save [2,4,...,1999999] // considering zero based indexing
